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[57] ABSTRACT 

A system for acquiring images during a medical procedure 
and using the acquired images includes a storage device for 
storing, for each one of a plurality of users of the system, or 
for each one of a plurality of medical procedures, or for each 
one of a plurality of input or output devices, information that 
indicates one or more processing operations to be performed 
on images obtained by an input device. A system processor 
responds to an identity the user who is cuirently using the 
system by performing processing operations on the obtained 
images and applying tiie images to an output device based on 
the stored infonnation that corresponds to the current user. 
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Sample Schema definition 

This sample is shown using RAIMA database languagot but can be applied to any 
generic database handler. 

record what { /Mield types V 

unique key long what_ id: /Mield key */ 

key long what_ owner; ' /'key of its owner 7 

char what_ name[32) /'field name V 

long what_ look /'it's appearance V 
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MANAGING INFORMATION IN AN 
ENDOSCOPY SYSTEM 

CROSS REFERENCE TO RELATED 
APPUCAnON 

This is a continuatioii-io-pait of U.S. patent applicadoa 
Ser. No. 08/040,633, filed Mai. 31, 1993 and abandoned on 
Sep, 6, 1995, the entire disclosure of which is hereby 
incorporated herein by reference. 

BACKGROUND OF THE INVENTION 

This invention relates to managing informatioii in an 
endoscopy system, and in particular to controlling the 
acquisition, processing, storage, and display of endoscopic 
images. 

Dining an endoscopic surgical procedure, the surgeon 
typically uses a video camera (or other suitable device, such 
as a video aithroscope) to c^ture images of the surgical site. 
The images are generally applied to a display device (such 
as a television monitor) for observation. In some cases, the 
images are stored on video tape (by, e.g., a VCR) or arc 
converted to digital files for storage in memory or on a disk 
drive. DifTcrcnt physicians often use the video equipment in 
different ways. 

SUMMARY OF THE INVENTION 

One general aspect of this invcntioD features a system for 
acquiring images during a medical procedure and using the 
acquired images includes a storage device for storing, for 
cadi one of a plurality of users of ttie system, information 
that indicates one or more processing operations to be 
pcrfomued on images obtained by an input device, and a 
processor that responds to an identity the user who is 
currently using the system by pcaf orming processing opera- 
tions on the obtained images and applying the images to an 
output device based on the stored information that corre- 
sponds to the current user. 

The stared infarmation also' indicates, for each user, a 
configuration of the input device and a configuration of the 
output device. During operation, the input and output device 
configurations arc established in response to the identity of 
the current user. The configuration of the input device 
includes the format in which the device produces the image, 
and the configuration of the ou^t device includes die 
format in which the device uses the image. As a result, the 
system can translate images obtained in one format (e.g., 
NTSC, RGB. Y-C. PAL. etc) to anotticr format for display 
by the ou^ut device, all acceding to the user-specified 
informatiOD in the storage device. 

The system may include a plurality of input devices and 
multiple output devices. The storage device stores the infar- 
mation for each inpMt device and each ou^ut device. During 
operation, the processor selectively processes images from 
the input devices based on the identity of the current user 
and the stored information. Likewise, the processor selec- 
tively applies the processed images to the ou^ut devices 
based on the identity of the current user and the stored 
information. 

According to aspect of the invention, the preference 
database stores, for each one of a plurality of medical 
procedures* information that indicates one or more process- 
ing operations to be performed on images obtained by an 
input device, and the processor responds to the identity of 
the medical procedure that is currently to be performed by 
performing processing operations based on stored informa- 
tioD that corresponds to the current medical procedure. 



\o,m 

2 

According to another aspect invention, the preference 
database stores, for each one of a plurality of users or each 
one of a plurality of medical procedures, information that 
indicates a script of designated images, and the processor 

5 responds to the identity of the cunrcnt user or the identity of 
a medical procedure that is currently to be performed by 
causing a "script" of images to be captured based on the 
preference information in the database. 

The information indicates, for each user or medical 
procedure, a sequence of images that are to be obtained 
during the procedure. In one embodiment, during operation, 
the cuaent user is prompted to obtain the images in the 
sequence. This allows the user to follow pre-established 
saipts (stored as directed by the user) to perform tours of the 
surgical site according to the user's preference or according 
to the requirements of the current surgical procedure. 

According to another aspect of the invention, the prefer- 
ence database stores the preference infonnation as a plural- 
ity of linked records. A first one of the records identifies a 

20 ^yP^ information, and a second one of the records 

contains data associated with that type of information. 

According to another aspect of the invention, the prefer- 
ence database stores, for each one of a plurality of users or 
medical procedures, respective preference information that 

2^ indicates one or more visual rq>resentations that are pre- 
ferred to be applied to the output device or devices together 
with the images obtained by the input device or devices. The 
processor combines the one or more visual representations 
with the images obtained by the input device based on the 
preference information in the preference database and 
applies the images to the output device or devices together 
with the one or more visual representations. 

The one or mere visual representations may Include text 
that is anti-aliased for in^oved appearance, graphical 

35 objects, or an additional image or images that may be 
obtained fiom an additional input device. The visual repre- 
sentation may possess the attribute of translucency. In ca- 
tain embodiments the visual representations are prompts for 
pron^ting a user to capture images in a desired sequence. 

^ The processor may maintain an aspect ratio of the visual 
representations with respect to the images as the images arc 
scaled in size, and may noaintain the relative appearance of 
the one or more visual rqiresentations with respect to the 
iiiuiges as the format of the images is changed. The proces- 

45 sor may store the one or more visual representations in 
overlays in order not to affect the images. 

In certain embodiments the visual representations are 
additional images the processor caused to be stored in a 
memory upon receipt of a save command during a medical 

so procedure, and the processor may cause the ou^ut device or 
devices to display the images as a composite of postage 
stamp size images or as a sequence of full size images. The 
processor niay be configured to invoke image processing on, 
or to delete, images selected by the user from among the 

55 images applied to the output device or devices. 

According to another aspect of the invention, the prefer- 
ence database stores, for each one of a plurality of input 
devices or output devices, information that indicates one or 
more processing operations to be performed on the images 

60 obtained by the particular input device or to be applied to the 
particular output device. The processor responds to an 
indication of the identity of the particular input or output 
device by selectively paforming processing c^erations on 
the images based on the information in the preference 

65 database. 

According to another aspect of the invention, the prefer- 
ence database pre-stores. for each one of a plurality of users 
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or medical procedures, respective preference ioformation 
that indicates a preferred page layout of images. The pro- 
cessor receives an indication of an identity of a user or 
medical procedure and causes a plurality of images obtained 
by the input device to be arranged in the page layout based 5 
on the preference infonnation. The preference information 
may indicate a preferred script of designated images that the 
processor causes to be captured based on the preference 
informatioa, the script of images comprising the images to 
be arranged in the page layout. lo 

According to another aspect of the invention, the proces- 
sor performs a zooming function for increasing size of the 
images on the output device or device, and the processor 
pecforras the zooming function concentrically about a cen- 
tral point of the images. By performing the zooming func- 15 
tion concentrically about a central point, rather than, for 
example, an upper left hand comer as in certain text pro- 
cessing systems, the processor accommodates the human 
tendency to focus on the center of an image. 

According to another aspect of the invention, the prefer- 
eoce database pre-stores, for each of a plurality of users, 
respective preference information that indicates a point 
about which the user prefers to have a zooming function 
performed^ The processor performs the zooming function 
concentrically about the point indicated by the preference 
information. 

According to another aspect of the invention, the proces- 
sor enhances visual quality of the images by histogram 
adjustment of the images, and adjusts the enhancement of 
visual quality based on the preference information corre- 
sponding to a particular user or medical procedure. 

According to another aspect of the invention, a still frame 
buffer is provided that tenqwrarily stores an image obtained 
by the input device, and a memory is provided that stores 3, 
images obtained by the input device on a long-term basis. 
When the processor receives a freeze command it freezes in 
the still frame buffer an image obtained by the input device, 
and when the processor receives a save conomand it saves a 
frozen image in the memory. ^ 

In certain embodiments, when the processor receives a 
live command it causes the output device to display live 
images obtained by the input device. The processor can 
toggle between frozen and live images until a desired image 
is frozen and then saved in the memory. 44 

According to another aspect of the invention, the prefer- 
ence database pre-stores, for each of a plurality of users or 
medical procedures, respective preference information that 
indicates a preferred conversion between a format in which 
the input device obtains the images and a format in which 50 
the ou^ut device communicates the image data to the 
medical practitioner. The processGo: translates the images 
between the two formats based on the preference informa- 
tion. 

According to another aspect of the invention, the proces- 55 
sor performs dynamic mathematical analysis of image data 
received from the input device. Based on the dynamic 
mathematical analysis of the image data, tlie processor 
automatically converts a format in which the input device 
obtains the images into a format in which the output device eo 
communicates the image data to the medical practitioner. 

According to another aspect of the invention, the proces- 
sor reduces interiaccd noise on an interlaced video display 
device by reducing motion artifacts due to motion that 
occurs between acquisition of fields of video that ooiiq)rise 65 
a video frame. At least one pixel in at least one line of a first 
field of video (preferably two pixels in two respective lines 
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of the first field) is compared with at least one pixel in at 
least one line of a second field of video that is interlaced with 
the first field on the video display device. The line or lines 
of the first field are in close proximity to (preferably adjacent 
to) the line of the second field. The pixel or pixels in the first 
field are in dose proximity to (preferably adjacent to) the 
pixel in the second field. The processor dctcrnaining whether 
a difference between the value of the pixel or pixels in the 
first field (preferably the average of two pixels in the first 
field) and a value of the pixel in the second field exceeds a 
fixed value. If the difference between the values of the pixels 
exceeds the fixed value, the processor replaces the value of 
the pixel in the second field with a value closer to the value 
of the pixel ot pixels in the first field (it is preferably 
replaced with the average of the above-mentioned two pixels 
in the first field). If the difference between the values of the 
pixels is less than the fixed value, the processor maintains 
the value of the pixd in the second field. These steps are 
performed for additional pixels in the above-mentioned line 
or lines of the first field and the above-mentioned line of the 
second field, and for additional lines of the first and second 
fields. 

As described in more detail in the detailed description, the 
invention provides numerous other features and offers sev- 
eral advantages. Among them are: 

1. a standardized platform for endoscopic video systems 
that allows mcH-e r^id evaluation and improvement of 
hardware and functionality. 

2. single acquisition and stcarage of still and live images 
from the operating room or endoscopy suite. 

3. digital enhancement of acquired images fox presenta- 
tion in printed and film formats. 

4. annotation of acquired images with text and graphics. 

5. formatting and jM-oviding conqwsites of multiple 
images, images and stock images, or drawings in a 
manner that provides spatial, logical or other context 
that improves the communicating value of the acquired 
images. 

6. a stand-alone database application that allows indi- 
vidual doctors to maintain and retrieve images and text 
data. 

7. an interface that allows existing office or hospital 
databases to manage image and text information relat- 
ing to endoscopic applications. 

8. allowing existing endoscopic database and endoscopy 
suite management systems (such as Siscope and others) 
to integrate imaging into their applications. 

9. in^roving the reliability and dependability of endo- 
scopic image and text data by facilitating sinq}le and 
reliable systems for integration of the data with physi- 
cian and facility work flows and hardware constraints. 

In addition, the invention iniplements a modular archi- 
tecture that allows inqirovement by upgrade instead of 
replacement. Regulatory requirements can often be satisfied 
by a modification to an existing filing, as opposed to a dc 
novo filing. 

Other features and advantages of the invention will 
become apparent from the following detailed description, 
and from the claims. 

BRIEF DESCRimON OF THE DRAWINGS 

FIG. 1 is a block diagram of a system for managing 
endoscopy information according to the present invention. 

FIG. 2 shows one possible configuration of the system of 
FIG. 1. 
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FIGS. 3 and 4 shows details of a subsystem used in FIG. in the preference database include identificatioD of available 

2. input and output devices^ the scan rate and resolution of 

FIG. 5 shows two subsystems of FIG. 2 connected in video input and video display devices, video gain, offset, and 

parallel for so-called **picturc-in-picturc" display of images. color parameters, light level reference for comparison and 

HG. 6 illustrates a procedure for c^turing and storing ^ adjustment of light level, the format with which images 

images in the system of HG. 1. obtained during a tour (described below) are to be displayed, 

« . tt. J * . . . and the format with which images are to be displayed on a 

FIG. 7 shows a W procedure for capturmg images in ^^-^^ 

the system of FIG. 1 according to a predefined script ^ . ^ T ^ . . , . ^ ^ . ... 

^. . ... . V Awidevanetyofunagemput devices 16 can be used with 

HG. 8 is useful m understanding how the system of FIG. lo endoscopic procedure management system 10. For example, 

1 formats piintcd outputs of images. ^^^^ ^^^^^ 1^ -^^^^ endoscopes, cameras 

FIG. 9 illustrates a procedure for annotating images (including endoscopic video cameras and other types of 

obtained by the system of FIG. 1 with text and graphics. video cameras), video cassette recorders (VCRs), X-ray 

HG. 10 is useful in understanding a character recognition scanners (which convert X-ray films to digital files), digital 

feature of the system of FIG. 1. X-ray acquisition ^paratus (which convert X-ray photons to 

FIG. 11 shows a procedure implemented by the system of ^ digital bitstream or file), fluoroscopes, CT scanners, MRI 

HG. 1 to convert the video format of an input image to a scanners, ultrasound scanners, RS170 video signal devices, 

different video format for use by an output device. ^cd devices, and other types of scanners (handheld or 

HG. 12 is useful in understanding the operation of a otherwise), 

remote control feature of the system of FIG. 1. Likewise, several different types of image output devices 

FIG. 13 shows a procedure peifOTmed by the system of 18 can be connected to system 10. Exan^es include cathode 

FIG. 1 for producing sUdcs from the acquired images. displays, television monitors, LCD panels 

T7ii-i u I. *!. J • i_ ji- L-^ x_ r and screens, EMM memory, and disk memory, digital film 

FIG. 14 shows the device handling architecture of the ^ . * • * / - * * j - 

system of FIG 1 recorders, postscript color pnnters, raster-dnven film 

™« ' ^ , . . . . recorders, and faster-driven printers. The displays may use 

HG. 15 lUustrates the use of umversal comiectivity ^^e RS170 format (i.e., RGB, Y-C, OTSC, PAL, or the PAL 

devices (that use, e.g., the PCMCIA standard) m the system equivalent of Y-C), or alternatively may be non-RS170 

ci FIG. 1. devices (interlaced and non-interlaced display devices). 

HG. 16 is useful in understanding the operation of a ^s described in detail below, visualization input and 
preference database of the system of FIG. 1. ^ ^^^^y subsystems connected with the personal computer 

FIGS. 17 and 18 illustrate the storage of acquired images 12, whether internal, or external, arc supported by defined 

in an image database of the system of FIG. 1. communication protocols (e.g., horizontal and vertical 

FIG. 19 is a flow chart that shows the op^ation of the synch, interlaced vs. non-interlaced display, scan speed and 

system of FIG. 1 from the standpoint of the user. spatial and color resolution) that are stored in the database. 

FIG. 20 shows modules of a software program that In addition, personal computer 12 also receives data and 

controls the overall operation of the system of FIG. 1. commands from one or more data input devices 22. 

FIG. 21 is usefvd in understanding the structure and Data input devices 22 can include computing devices such 

organization of the databases used in the system of FIG. 1. as laptc^ and palmtop con^iuters, pen-based command 

4Q systems, or computer networks. Hie data and commands 

DETAILED DESCRIPTION fj-Qj^ such computing devices function primarily to allow 

Referring to HG. 1, endoscopic procedure management entered outside the operating room to be used within die 

system 10 employs a personal computer 12 that executes a operating room. For example, a surgeon's assistant may 

stored program 14 to configure and manage all devices used ^^^^ identification infcnnation about a day's work on such 
by a physician during endoscopy. In particular, personal 45 ^ computing device to load that information into endoscopic 

computer 12 receives images generated by one or more procedure management system 10. In this way, operating 

image input devices 16, processes the images according to room data entry can be minimized, 

the preferences of the physidan performing die endoscopic Data input devices 22 can also include prpgrammable 

procedure, and transmits the processed images to one or input devices (such as infrared remote control units), voice 
more unagc output devices 18 for display or storage. 50 recognition systems, and pointing devices such as a 

Personal conqMiter 12 obtains information concerning the keyboard, mouse, trackball, airmouse, light pen, touch 

physician's preferences about the processing, display, and screen, and tablet These programmable input devices opcr- 

storagc of the images firom a preference database 20. As ate by a software-mediated control step. In particular, die 

described in more detail below, preference database 20 is programmable input devices generate generic commands, 
organized by physician — that is, the preferences of each 55 ^^d die response of endoscopic procedure management 

physician recognized by system 10 as to the image process- system 10 is stared in software. Use of these programmable 

ing and the configuration and operation of image input input devices can initiate a simple or complex sequence of 

devices 16 and image output devices 18 are stored in events. 

preference database 20 according to the identity of the All of these data input devices 22 allow the physician (or 
physician and according to the various procedures per- 60 another user located, e.g., in the sterile field) to manage the 

formed by the physician. As a result when a physician logs operation of system 10, as described in detail below, 

onto system 10 (as described below) and identifies the Personal computer 12 can also be connected (cither 

endoscopic procedure that he or she is to perform, host directly or via a network or modem) to one or more 

computer 12 configures image input devices 16 and image unrelated databases 24 to receive patient and other informa- 
output devices 18 and controls the processing of the 65 tion. Examples of databases 24 include hospital information 

obtained images according to the information stored in systems and local databases located in the physicians* 

preference database 20. Examples of inf<Hmation contained offices, patient information databases, external office 
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databases, external hospital databases., external database 
conversion utilities, and external high-capacity storage 
devices. 

Numaous data output devices 26 are also connected to 
personal coic^uter 12 for receiving data concerning the 
operation of system 10 (e.g.. reports). Personal con^uter 12 
also sends data to one or more of databases 24 for storage. 
Examples of data output devices 26 include laptop, palmtop, 
and pen-based computer systems, printers, disk drives, 
PCMCIA memory cards, modems, and network communi- 
cations devices. 

Referring also to FIG. 2, one possible configuration of 
system 10 is shown. Personal computer 12 includes a host 
processor 30 and a scries of subsystems or adapters 34a-34p 
(hereinafter generally referred to as 34) each of which is 
connected to host processor 30 by a bus 36 (if the configu- 
ration of personal computer 12 so dictates, bus 36 is replaced 
by a backplane). If needed, one or more subsystems 34 can 
also be connected to host processor 30 or to each other via 
a local bus (not shown). 

Host processor 30 provides an industry standard platform 
that suppUes a common power supply, a confuting fiinctioD 
(CPU 38). memory and storage devices 40, mechanical 
support and cooling fiinctions 42. Subsystems 34 gain 
access to any or all of these services via bus 36 in a 
motherboard in^)iem£ntation (or the baclq>lane in a back- 
plane implementation). CXhcr sy5tcn:is outside the physical 
housing of personal computer 12 can gain access via sub- 
systems 34 that externalize these functions from personal 
computer 12. 

Host processor 30 is a 486-66, Pentium, or 686 based 
machine; alternatively, host processor 30 may be imple- 
mented as a workstation. Standard ISA bus architecture is 
used, but EISA or PCI architecture or others can be selected 
instead. In the motherboard configuration, the memory and 
CPU and expansion slots are on a single board. In the 
baclqplane configuration, power and communication occur 
via the baclq)lane, and CPU 38 and memory 40 phig into the 
baclq>lane Uke an adapter. 

Host processor 30 contains a special version of a BIOS 
(Basic lapnt/Output System), located on a ROM chip, that 
specifies the setup parameters of a generic personal com- 
puter in such a way that operation of a system storage disk 
and connected devices can be restored despite failures that 
would render a standard personal con^uter useless. The 
CMOS settings on a generic personal computer are generic, 
and the setup parameters are held in CMOS, which depends 
on battery backup and system integrity to persist. If a 
peripheral shorts, or the backup battery goes down, at least 
some of the system settings might have to be re-entered 
before the system can be used. By restoring the CMOS with 
the ROM, even custom setups of the PC can be re-entered 
by software so that the PC can be fully recovered from the 
majority of problems that could bring a PC down. 

Subsystems 34 and host processor 30 communicate in a 
bi-directional fashion over bus 36. Each issues and receives 
commands, status and other information. Host processor 
software control 14 integrates and regulates the flow of 
infoiroation, and hence the operation of subsystems 34 and 
the input/output devices. Communications that conform to 
platform standards can be handled by bus 36. Specialized ot. 
nonstandard conuntmication can be handled by local buses, 
such as a local video bus (not shown). A local bus provides 
specialized cormnunication of limited scope. Overall func- 
tion of die local bus can be governed by communication that 
occurs over bus 36. 
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The communication afforded by bus 36 allows for soft- 
ware control of the specific functions of system 10. Software 
control of specific functions can be based on data entered 
into the preference database, operator instructions received 
via data input devices, or on feedback from the subsystems. 
For example, preference database 20 contains information 
regarding video gain and offset, image format preferences, 
and text data required for display of Images obtained during 
endoscopy. Settings or user preferences can be accoirmxo- 
dated and tailored to the (typically different) spediications 
' of each physician. By specifying a particular procedure or a 
particular surgeon it is possible to cause preferences asso- 
ciated with the identity of the surgeon or the procedure to be 
loaded. Id addition, the operator can modify the preference 
settings by direct instructions communicated via a system 

1^ input device. Furthermore, the system itself can modify the 
prefa*ence settings. In system-mediated modification, the 
system responds to specified parameters based on the state 
of the system or a portion of the system. For example, the 
system allows for a histogram, or other mathematical pro- 

20 cess to be performed on an image. Based on the result of the 
mathematical process, the system software can send com- 
mands or data to the camera or video signal source. The 
commands result in optimization of tiie acquired video. 
Other system-generated input, such as video level, color 

25 composition, or most recent operator interactions, can be 
used to specify a system response. Thus, preferences can be 
modified by the operator, or dynamically by system 10. 
Thus, the system responds to database entries (procedure or 
user ID), calculated entries (histogram of an image, video 
level via defined relationships stored in the preference 

^ database), and user input (keyboard, mouse, or other point- 
ing device). 

As discussed below, multiple control devices are sup- 
ported. An infra-red, programmable handheld remote con- 
3j trol 48 provides operator interaction from a sterile field. 
Hard-wired switches connected by wires to the operative 
field can also be used. The system also provides for multiple 
inputs from touchscrecns, mouse, light pens and other 
relative and absolute pointing devices 58. 

The system is structured to be platform independent. The 
software provided for the system is organized so that por- 
tions of the code that are hardware dependent can be easily 
replaced. As CPU and adapter subsystems are iii^oved or 
changed, a means is provided to allow simple integration of 

45 the new devices. Identifications of input and output devices, 
as well as specifications, are stored in preference database 
20, and this stored information controls device-specific 
ou^uts. For example, 24 bit color images can be processed 
to display on a 64 color gray scale device based on the 

jQ database preferences stored in database 20. 

The platform independence is related to separation of 
hardware-dependent code from hardware-independent code. 
In one embodiment the system runs under DOS, but with 
simple modification could run under Windows, OS/2, UNIX 

55 or otiier operating systems or hardware platfcnns. This is 
accomplished by separating generic file system calls, calcu- 
lating functions, display operations (blitting) from operating 
system or hardware dependent functions. In this way, a 
change of the database utility, driver utility or any other 

60 segment of the program can be done without altering the 
main body of the code. 

As is discussed below, the software system includes 
heuristic systems that allow multiple complex functions to 
be controlled automatically or with minimal and simple 

65 operator input 

The host 30/subsystem 34 architecture allows for both 
standardized communication therebetween via bus 36 and 
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the sharing of resources (such as memory and processing intezpreted by the video subsystem to produce images on a 

functions). Subsystenas 34 identify themiselves to host pro- display device. The system has a control mechanism, for 

cessor 30, send queries to and respond to queries from host example, RS232, with which the host can issue commands 

processor 30, and provide signal and control outputs to host and receive status from the camera system as described 

processor 30. 5 above. 

Communication occurs over bus 36 between host proces- Remote control adapter 34c interfaces with infrared 

sor 30 and subsystems 34 and between the subsystems 34 remote control 48, which includes a transmitter and a 

themselves. As can be appreciated from FIG. 2, inputs can receiver. The transmitter is a handheld unit that emits 

be applied to personal computer 12 either through the infrared radiation in a predictable manner, based on Inslruc- 

subsystcms (e.g., subsystems 34a-34g) or directly to host 10 tions stored within the unit. The Receiver detects and 

processor 30. likewise, output signals and data are produced interprets infrared emissions of the transmitter, and produces 

by both host processor 30 and the subsystems (e.g., sub- machine readable commands that are communicated to the 

systems 34f-34p). CPU, which colI^)ares the commands to taxonomy con- 

FIG. 2 illustrates several different types of subsystems 34. tained in preference database 20 and executes the command 

Camera subsystems 34^1, 34b provide host interfaces for a based on the information contained in the preference data- 
camera 44 and a video endoscope 46. Remote control 

adapter 34c interfaces a remote control unit 48 (infrared or Voice Control Adapter 34d accepts and processes audio 
otherwise) to host processcs^ 30, The physician can control input, from a roicroj^onc, and performs interpretation, 
system 10 using voice commands that are relayed from resulting in machine-readable signab that are transferred to 
voice pickup 50 to host processor 30 by voice control the CPU. The function of the voice control ad^tcr is to 
adapter 34d. A fluid/gas monitor 34<r relays signals from a translate spoken words or speech into machine-readable 
fluid gas sensor 52 in the sterile field to host 30, Laptop, input commands. The machine-readable commands can be 
palmtop, and other computing devices 54, and pointing used as a means to implement controls. Alternately, the voice 
devices communicate with host processor 30 via suitable recognition can be used to enter text data into the database, 
adders 34/ PCMCIA devices 56 (discussed below) are piyij/gas monitor 34e interfaces with fluid/gas sensor 52, 
interfaced with host processor by PCMCIA ad^ters 34^. which is, in one embodiment, an insufflator, i.e., a device 
Other input devices 58 (such as a keyboard and pointing that controls flow andpressure of amedical device into body 
devices such as a mouse) are connected via ^edfic inter- cavities. It communicates with the CPU by RS232 or a 
faces directiy to bus 36. ^ device-specific comiminication protocoL The CPU trans- 
Video subsystems 34/, 34* provide interfaces with a CRT latcs database preference values to commands that configure 
display 60 and a television monitor 62, The level of light the pressure and flow parameters on the device. The device 
provided in the surgical field by an illuminator 64 is con- issues status and error messages to the CPU via the same 
trolled by light source subsystem 34/. A VCR subsystem conmiunicating means. 

Mm supplies image signals to a VCR 66. The operation of 3^ Adapter 34/ is a device-specific ad^ter for receiving 

devices used m the surgical field, sudi as fluid supply 68 and iofoimation from, or transmitting information to other 

one or more medical devices 70 (e.g., surgical tools such as con^ut^s, i.e., main-frame hospital computers, personal 

shavers, abraders, end cutters, etc.), are controUed via fluid computer networks, handheld con^)uters. This interface is a 

management subsystem 34rt and medical device subsystem means for exchange of text and image data. 

340, respectively. Oth^output devices, such as printers and ^ ^^^^ 3^ ^ ^^^^ ^ ^ 

^r^R'^t Z^^^^^ configuration tointerfacl to a wide variety of devices that 

more PCMCIA ad^ters 34b, while some output devices 74 za^^h^i^a ™*™ v *u • «*i * a 

u *TT u sre identified to the host system by the msertion of the 

can be connected directiy to bus 36, , Tw^Tyr^xMmA a / * i_ j 

^ device. The PCMCIA adapter can accept specific hardware 

Preference database 20 can also be connected to host to perform modem, network, storage or other functions. The 

processQr30viaaPCMCIAadapter34f,APCMCIAadapta- 43 CPU side of the interface is standard, and confoms to the 

34/r IS used to connect host processor 30 to units 76 that PCMCIA standard for power and data exchange. 

provide other services. .Examples of other units 76 include j^^^ ^^^^^^ ^.^^^ ^ 

T^^^^'^l"'' '^w.*^!!:'^ memories (such as nionitors and CRT displays, 
so-called ^ashmemones ), disk dnves, and network ad^- 

ers (for wire, wireless, RR or optical networks). Any device 50 "^^^^ subsystems 34; and 34* are set forth 

that conforms to tiie PCMCIA connectivity standard (or any connection with FIGS. 3, 4, and 5. 

other suitoble standard, such as JEIDA) can be connected to ^S^^ source subsystem 34/ processes signals generated 

personal computer 12 in this fashion. In addition, any by the video subsystem, and outputs an electrical potential 

suitable port of hostproccssOT 30 (e.g.. serial, parallel, SCSI, ^ adjust the intensity of light provided by the illumi- 

etc.) can be used to provide I/O capability. 55 Light intensity is controlled by closing an aperture, or 

Camera subsystems 34a and 34i? utilize a PC standard <lecreasing lamp supply voltage, 

adapter card configuration containing circuitry to process Details of VCR subsystem 34m and VCR 66 are set forth 

electrical signals generated by a device responding to elsewhere in this ^plication. 

changes in available light for example a charge coupled Fluid management subsystem 34rt interfaces with fluid 

device. The circuitry supplies power and synchronization for 60 supply 68, which is a pump and suction apparatus to provide 

the image acquisition device, and provides a means to control of volume and pressure of irrigation fluids intro- 

communicate with the host system. The host system is able duced into body cavities. The system senses and reports 

to download alignment parameters to the CCD controller, pressure, flow, and status. The system interfaces with die 

and interpret commands and sutus returned &om the sub^ CPU and preference database via a RS232 protocol, or 

system to the CPU, Camera 44 and video endoscope 46 are 65 device specific protocol. Database preferences for pressure, 

self-contained units that acquire an image, and produce a flow and operations are communicated to the device, and 

standard video or digital signal that can be decoded or stams is returned by the device to the CPU. Pressure and 
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flow parameters are controUed based on user Identification, 
procedure, or a combination of user and procedure. 

Mcdicid device subsystem 34o is a generic interface to 
conform to the evolving medical device control standard^ 
whereby other pumping, light, or other medical device can 
be operated via a standard interface communication between 
(he CPU and the device. Hie system supports standard or 
non-standard interfaces including serial, parallel, SCSI, 
GPIB or other protocols. 

PCMCIA adiiptCTS 34p interface with printers and film 
recorders 72, which can be controlled via a parallel, SCSI, 
or GPIB interface, coupled with a device-specific software 
driver. As in all devices in the system, the device-specific 
driver translates generic conunands from the system into 
specific conunands understood by the device. The device 
(^ver also interprets device specific-status reports, transla^ 
ing them to specific or generic reports and requests that are 
interpreted by the CPU, and result in continued operation, or 
error reporting. 

Printers and film recorders 72 can include a post script 
printer. Conforming to Adobe post script level 2 page 
description language, a text file containing information 
sufficient to allow rendering of the image is transmitted to 
the printer. The description contains all necessary data to 
allow the raster image processor to size, scale, and position 
text and images. Data is transmitted in a device-dependent 
manner as either binary information or ASCII diaracter 
strings. 

In connection with the use of film recorders, raster infor- 
mation is transfarmod to the appropriate resolution by a 
bicubic interpolation routine. Sizing and gamma con'ection 
is done using standard algorithms. Hie image is separated 
into red, green, and blue components. Using device-specific 
look-up tables, the color-spedfic intensity of each data point 
is translated into a film-specific exposure time. The data is 
downloaded to the imaging device that accomplishes expo- 
sure of the film in a red, green, and blue cyde. 

Printers and film recorders 72 can include a 24-bit true- 
color VGA device. Raster information is presented to one of 40 
several supported display devices by translating pixel infor- 
mation and CPU instructions to the display chip, using 
protocol provided by the chip designer, manufacturer. 

Referring to FIGS. 3 and 4, details of video subsystem 34j 
arc shown (it will be understood that video subsystem 34fe is 
identical to subsystem 34/). Video subsystem 34y resides on 
full length PC card, and contains the necessary processors 
and memory to allow real-time gain and offset control, and 
matching of multiple inputs to multiple ou^ut timing speci- 
fications for display, pan, zoom, video overlay and digital 
signal processing. The system contains a resident, read only 
memory BIOS tfiat allows this subsystem to operate in a 
limited fashion as long as power is good, even if other 
components of the host system 10 fail. 

The functions of video subsystem 34y arc performed on 
the board in response to software commands to the board. 

The multiple functions of video subsystem 34/ include: 

1. Encoding and decoding RGB, NTSC, Y-C and PAL; 

2. Synchronization functions: acquisition synch, display 
synch, pixel dock control and synch conversion; 

3. Digital signal processing (DSP); 

4. Image manipulation operations such as zooming and 
panning; 

5. Drawing graphics such as boxes and lines; 

6. Control of video memory (i.c., in, out); 

7. Controlling aspect ratio; 
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8. Select between display of: 

(a) the image that currently resides in the frame bufi'er, 
or 

(b) the live image; 

9. Definition of input and display device parameters; 

10. Acquisition and display look up table controls; 

11. Acquisition and display weighing, adjustment; 

12. Color extraction and object recognition functions; 

13. Masking functions; 

14. Video display/overlay keying; 

15. Convolution; 

16. Management of prograimmble events; 

17. Mathematical operations on video signals; 

18. Inter-image operations and manipularions; 

19. Histogram extractions from images; 

20. Operations on defined areas of interest; 

21. Morphologic manipulation of image information; 

22. Extraction of a profile of pixels for a defined sample 
range; 

23. Control of pixel flow between host and subsystem 
memory; 

24. Extraction of ttie run-length of an image; 

25. Controlling the transfer of host and adapter mcmcffy 
buffer lines; 

26. Controlling the range and offset of analog-to-digital 
converter; 

27. Write and position text; 

28. Analog-to-digital conversion; 

29. Digital-to-analog conversion. 

The majority of the functions performed on the video 
subsystem 34/ are controlled by the Texas Instrument 34010 
chip. Other chips and functions can be added including DSP 
chips that can perform some of the histogram functions 
(described below) or other calculating functions more rap- 
idly. 

Many of these functions may be reproduced in software 
for still image processing and real-time processing in virtual 
mcraoiy of host processor 30. 

As shown in FIGS. 3 and 4, video subsystem 34/ receives 
an image 80 (to be displayed, e.g., on CRT display 60) in any 
suitable format (sudi as RGB, NTSC. Y-C, PAL, or PAL 
Y-C). The image is applied through an analog signal con- 
ditioner 82 and a range and offset controUo^ 84 to an 
analog-to-digital converter 86. The digitized image is 
applied to a synchronization module 88, which also receives 
as inputs a camera synch signal 90 (and possibly other 
external triggers 92). Synchronization module 88 can also 
generate synch signals for one or more input devices (such 
as cameras). The synchronized image signal 94 produced by 
module 88 is passed to graphics processor 96. 

Graphics processor 96 performs the graphical functions 
descril)ed above (several of which are listed in box 98) on 
image signal 94. The image produced by graphics processor 
96 is sent to frame huffier memory 100 for subsequent 
display. Graphics processor 96 also returns information 
about the live, processed image to host CPU 38 via bus 36 
(or via a local bus within personal con^uter 12). CPU 38 
responds to this information, and to the display preferences 
(from preference database 20) of the particular physician 
who is using system 10, by dynamically adjusting one or 
more display parameters of the live image in the video 
subsystem. The system can also send instructions and setup 
parameters to the video input device. 
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Histogram processes are standard mathematical distiibu- In addition, execution of the zoom function adjusts an 

b'ons of the digital values representing each of the red, green electrical potential output (not separately shown) that allows 

and blue components of each pixel. Histogram and distri- any external device to interpret the potential and adjust light 

bulion functions are resident in the graphics processor 96. In intensity so that the light level is appropriate to the magni- 

responsc to a command from the CPU 38, via bus 36, the 5 fled field of view. The electrical potential is related to the 

graphics processor will return a table of values correspond- luminance or brightness of the reduced field of view caused 

ing to the output of the histogram operation, or a selected set ^ ^oom, as raposed to the entire incoming video signal, 

of values as spedfied. These functions are part of the ^^^^ appropriate Ught level for the area being visu- 

standard funcUonahty of the TMS 34010 processmg chip. f r* f & & 

Some or all of these functions can be reproduced on the CPU « • • j/- 41 ^ j» j- ji. 1 ^ 

using software emulation of the grapt^processor function. , » an image is saved (i.e "captt^ed,; ^ 

Software program 14 compares diTrcsilt of the specified f*mng zoommg, a tag is placed m the im^^ 

operation returned by the graphic processor 96 to a tax- tag alerts an output device that uses the image file to di^^^^ 

onomy of values stored in the preference database 20, and the image as zoomed, i.e., in the same manner m 

executes die response corresponding to the values. The which tiie image was displayed to the user at the time of 

taxonomy of values can correspond to user, system, or other capture. Alternately, the device may ignore the tag, and may 

designations that can be modified by the user. P"^t the image without modification, if this is so specified 

In response to histogram calculation on an area of the iu the preference database, 
image, the system can download new camera (input source) Ih the non-zoom, or 1 :1 mode, area of interest processing 
configuration tables. This results in concction of the input can be used to generate a light source control potential that 
signal, as compared to enhancement of the signal that occurs 20 is weighted for a defined area that is a subset of the entire 
on the video subsystem card in the digital realm. viewing area. This area of interest can be dynamically 
£xanq>les of the processing performed by CPU 38 on the changed by an algorithm in the software. Area of interest 
image include masking 102, keying 104, VGA-, video processing is in^lemented by selecting a defined set of 
source-, or CPU generated-ovedaying 106, and output lev- pixels that is a subset of aU pixels that comprise a given 
eling 108. 25 frame. In one implementation, a circular or square sub- 
Masking involves selection of portions of an image by sample of pixels is selected by an instruction from the CPU 
color or edge detection. Masked areas of the image can to graphics processor 96 that specifies the range of pixels, as 
either be the target for further processing or excluded from well as the X,Y location of the offset of the selected pixel set 
further processing. This same selection tedmique is also used to perform 
Keying involves sdectioa of areas of the image by color 30 con^arisons of several selected areas of the image. For 
attribute. The area of the image with a designated pixel or example, in an arthroscopic procedure, the subsample area is 
color value can be operated on (e.g., replaced with other defined as a circle in the center of the image, with a diameter 
pixels, changed to another pixel colcr or value, etc.). Keying equal to the smallest dimension of the imaging area. Image 
operations arc performed by sanqiling areas of the image processes, and all image functions of the graphic processor 
designatcdby a preference stored in the database or program 35 96, or software emulation of these functions, can be per- 
itself, sampling the area for digital color level in R,G3 or formed on die selected area in the same manner as they are 
other suitable mode, and selecting or excluding pixels from performed on an entire image. 

the image matrix based on the selection criteria. For System 10 also provides for manual, uscr-p-omptcd con- 

exanq>le, the black border around an arthroscopic image can trol of gain and offset, specification of multiple values for 

be excluded from image processing operations by san^ling 40 gain and offset in preference database 20, and a real-time 

the black border, selecting it and excluding it from finlher process for automatic context and value-sensitive means for 

operations. control of gain and offset Automatic adjustment is accom- 

Overlaying involves displaying text, graphics, or other plished by mathematic processing of the signal in areas of 

images on top of the image. This method is used for display interest on the live or incoming video. In one 

of stams and some display of text 45 implementation, the individual RGB gain and offset param- 

Ih addition, CPU 38 adjusts (where required by the eters are reduced to user selected controls for brightness, 

preferences from database 20) such display parameters as contrast, and hue. The brightness, contrast, and hue func- 

display synch, pixel rate, and frame control (110). Ihe tions allow the user to adjust multiple video parameters in a 

resulting image information (consisting, e.g., of BLUE LUT, linear or non-linear fashion in a familiar, and simple manner 

GREEN LUT, RED LUT, and SYNCH) provides the output 50 The combination functions of contrast, hue, and brightness 

112 of video subsystem 34/. simultaneously adjust multiple gain and offset features in the 

The zooming function performed by gr^hics processor R-G-B domain, resulting in image changes that mimic the 
96 allows real-time images or still images to t>e magnified by adjustment of a television. The system may also contain a 
a user-specified value. The zoom can occur concentrically number of preset gain and offset groupings. The groupings 
about a point specified by the user or preference database. 55 can be related to specialty or procedure. Hiese presets may 
The usual specification for zoom concentricity is the central be accessed by sequential, single key offset adjustment 
point of the image, although any point may be specified. cormnands from the remote control The system provides a 
Image processing is done to allow the highest quality view standard, cr default setting that produces an output image 
of the magnified image. This is either a specific hardware that is the same as or similar to die input video signal The 
routine cr can be performed in software in a virtual frame 60 gain and ofi^et settings are implemented by modification of 
buffer. In this in^lementation, a zoom command issues a the input or ou^ut lookup tables (LUT*s), processes imple- 
hardware call to the graphics processor 96. The software call mented by the dedicated graphics processor, limits on the 
specifies the magnification value, as well as the x,y coordi- digital-to-analog ar analog-to-digital converters, or chip- 
nates of the location of the zoom, which in one implemen- specific instructions to the video encoding or decoding 
tation is the center of the image. The zoom occurs by pixel 65 hardware. 

replication, resulting in apparent magnification of the dis- For example, there may be several gain/offset combina- 

played image. tions that arc usefiil for a gallbladder. A single keypress on 
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the remote would cycle through the preset settings. The video subsystem H*. The video switch contains standard 
operating range and incremental change per press can be set circuitry for the combination of an analog video signal. In 
by specialty, surgeon, or procedure. This is considerably addition, a software-controlled switch toggles between inter- 
more flexible and functional than adjusting the video using position of the video switching apparatus, or direct input to 
switches or dials, which are impractical to use during a 5 subsystem signal communication from the primary video 
procedure. input source. When the switch is implemented, the input 

An inyortant feature of this system is monitoring for signal to the video subsystem will consist of the combined 

video signal. The software can monitor when the board of signal from both video sources. The composition of the 

the video subsystem has live video input. Lack of live video combined signal will be dictated by the video switch cir- 

input generates a message to the user to correct the defi- lo cuitry. The video processing functions of subsystem 34/ can 

ciency. be implemented only on the combined signal, as opposed to 

Another important feature is the direct video signnl each individual component signal corresponding to each 

bypass. Because the system is placed in the video stream, individual video input source. 

failure of the system could interfere with coir^letion of the Referring to FIG. 6, another function performed by video 

procedure. For this reason, a second path is provided for 15 subsystem 34/ is capturing images from frame buffer 100 

video dircctiy from the video source to the screen so that the (FIG. 4) and storing the images in memory, 

procedure can continue even if these other functions added Capture can occur by direct opcratOT interactions (via, 

by the system are lost e.g., remote control 48 or a pointing device). A sii^e button 

The architecture of video subsystem 34/ allows images to press can take a snapshot of the cunent information on the 

be displayed in a variety of formats according to the pref- 20 screen and transfer it lo a memory media as a digital file, 

crcnces of the physicians tiiat use system 10. For example. This system provides additional logic to allow toggle 

referring also to FIG. 5, a pair of video subsystems 34/, 34* between *treeze" and "live," until the desired image is 

are wire-ored together to produce a '*picture-in-picture" frozen. The image can then be saved to memory, by cxecut- 

ou^ut for display. Both images are applied to each video ing save command 120, Thus, the surgeon can elect to save 

subsystem 34;, 34*. Each subsystem 34/, 34Jt processes one 25 or discard any image. 

image and simply monitors the other image for synchroni- The freeze and c^ture functions can be controlled by any 

zation purposes. For example, video subsystem 34/ pro- data input device 22. One implementation allows control by 

cesses input 1 and monitors input 2, leaving it to video hardwired buttons on the camera 44 or infra-red handheld 

subsystem 34fc to process input 2 (and monitor input 1), remote control 48. The review function 122 displays all 

More than two inputs can be used, if desired. 30 images captured, either as a composite of many postage 

The image inputs may be either synchronous or asynchro- stait^ size images, or as a sequence of full size images. This 

nous. That is, video signal synchronization can occur at the allows the operator to be sure adequate images have been 

inputs to subsystems 34/, 34* via a system-generated synch captured. 

signal, or at the level of subsystems 34/, 34* in display The system allows the user to view all acquired images 

management by CPU 38. Host CPU 38 and program 14 35 that are related to a specified procedure. The images are 

control the output video, the asynchronous video signals to viewed as an array of images, reduced In size to allow 

a single output display. The wired-or connection provides a several images to be viewed on a single screen, 

current drive that adds the two image signals together. The The multi-image viewing function allows the operator to 

sizing, location, and priorities of the two images are review acquired images during or after the procedure to 

selected, switched, and defined based on the preferences set 40 verify that adequate images have been obtained. During a 

forth in database 20 and the sterile field controls (e.g., procedure, the review function is accessed by a keypress, 

remote control 48, FIG. 2) operated by the physician or other infrared remote control signal, or camera button function, 

user. Each signal input can be individually processed and All images acquired during the active, cs: relevant case arc 

controlled with the nmltiple image and signal processing displayed in an array of multiple images on a single screen, 

functions discussed above. The CPU commands for han- 45 The maximum number of images pcx screen and the size and 

dUng the multiple images are in addition to the conunands location of the image display arc designated in the program 

listed above, and are relayed to the subsystems 34/, 34* via setup and uso: preference database 20. Additional pages or 

jjus 3^ scrceas of images are added to allow review of all acquired 

In particular, priorities of location of images can be images when the number of acquired images exceeds the 

controlled by software-controlled placement of a "black*' or 50 maximum number of image display locations specified for 

••blank" signal, outside the value of pixels found in a typical the review screen. The usct can page forward and backward 

image in the output image. At the output stage, the video through the acquired images. Another operator signal causes 

signals are combined or added, so that the second (or third, tiic system to return to the live video acquisition mode or 

etc.) live video signal is added to the black or blank area of exit the acquisition mode. 

the main image. The resident image processor, which pro- 55 Thus, the system allows ttie user to rapidly review all 

cesses the signal for an image designated to occupy less than images stored during a procedure prior to completion of the 

thefullavailablcdisplay, can handle reduction of the image procedure. The system provides for rapid alternation 

display size. The reduction of display size can be accom- between a page of ^'postage stamp" miniature representa- 

plished by a pixel or line subsampling, or pixeVline com- tions of stored images acquired during the current procedure 

bination by a mathematical weighing of pixels. This results 60 and live image acquisition, 

in more accurate spatial and color representation in the After a procedure, images acquired arc presented in a 

image, even though the actual number of pixels used to similar multiple image per screen display manner. Images 

display the image is diminished, can be selected from all available images using the database 

Alternatively, a lower-cost implementation of picture in query mechanism, or by relationship to any criteria specified 

picture, or simultaneous display of two image input sources 65 by the user, or other programs employed by the user. When 

on the same display, can be accomplished by a software- used in this mode, a function is provided to allow the user 

controUed video switch applied at the input side of a single to select an individual image from the multi-image format 
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display. After sclecdoo, the designated image is displayed at 
full saeen scale and size. At this time, the user is presented 
with additional options for image mampulation, printing, 
annotating (adding text to the image), moving or copying the 
image to another location or file^ or deleting the image. 3 

The speed of transfer of images is important, as is the 
simultaneous aeation of a postage stamp size image to be 
used later for the review function and an image library 
browse function (described below). 

The frame buffer is transferred to a designated set of pages lO 
in mcmosy as planar pixels (one plane each of red, green and 
blue). All .operations on the pixels are performed in RAM. 
The postage stamp intiage is subsan^led by a rapid scheme 
that only performs the dual writes of designated pixels with 
TTiiniTnal CPU time. Disk write, which is the slowest of the 15 
functions, is performed by block writes, which are the fastest 
way to do this. 

Hie review function works as follows: When the com- 
mand to review is issued, the system looks at the current 
image directory and pr^ares space for one postage stas^ 20 
per image in tiie directory. The system pulls up the postage 
stamps and loads them to memory by reading the designated 
part of the image file by offset so that no calculation or 
reading other than the image is required The layout of the 
postage stands for display is specified by a form. This 25 
makes it fast. Once the page is configured in memory, it is 
transferred to tiie display device. The review function oper- 
ates quickly, thereby allowing surgeon to see what's already 
in memory, which is an important feature of the system. 

The same method is used to display images in the library 30 
browser function, but in the browser function, the image 
itself is used as a button that, when activated, results in 
di^lay of the full screen version of the image associated 
with the postage stainp. 

The size of the postage stazi^ size images and the layout 35 
of the browsing screen can be set by preference. In one 
embodiment, it is 3 rows of 4 image each. When the number 
of images stored exceeds 12, additional pages are con^sed 
and displayed 12 images at a time. 

FIG. 7 fllustrates a **tour" function performed by system ao 
10. Preference database 20 contains, for each physician who 
uses system 10, one or more ''scripts" 130 for designated 
captures of images. When the tour function is initiated (132), 
system 10 pron^ts the physician (134) to capture images by 
anatomic site. path(dogy or other designation. Proii^ting is 45 
pcrfomied via a gr^hical object or a text clue imposed on 
the displayed image that guides the operator to the proper 
position for capture. The physician has the opportunity to 
skip an image or acquire additional images during the tour 
(136). Upon the completion (138) of the script, system 10 50 
has captured andidentitied a series of images (140) spedlied 
in the script 

Multiple tours, i.e., pre-op and post-op tours can be 
designated (142). Pre-op and post-op tours advantageously 
show pathology and post endoscopic procedure results or 55 
status. If multqile tours are specified, system 10 selects (144) 
the scripts for the tours from preference database 20 in a 
logical sequence that conesponds to the natural fiow of the 
surgical procedure. The format of die printed ou^ut may 
identify the sites as pre- and post-intervention. 6o 

A special case of tour is invoked when images acquired at 
different times during a procedure are to be displayed on a 
single tour print format. In this instance, it is desirable to 
interrupt or resume a given tour. When a tour is configured 
such that it can be interrupted, sequential ^stait tour" com- 65 
mands will cycle through available tour scripts. For 
example, 3 tours can be attached to a given form, and 
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designated as interruptable. The first start tour command wUl 
initiate tour script #1, a second keypress advances to script 
#2, a third to #3, and a fourth start tour conmiand returns to 
script #1. If an image is present for a given prompt in the tour 
script, it is displayed to the screen. At this time, the user 
presses "skip" to advance to the next prompt, or "capture" 
to return to live video, discard the image associated with that 
prompt, and re-capture an image to be associated with the 
prompt. Altematiydy, the user can specify that re-activation 
of an incomplete tour will stop only at prompts that do not 
already reference an associated capture image. 

The available tour scripts arc related to the form in use 
when the tour function is initiated. Tlie printed output of the 
tour and its labeling are specified by the output format file 
called by the tour. The combination of the system prompts 
to the user and the logic used to prepare the tour script arc 
what give the system the ability to infer something about the 
content of the image for use in the tour formats where text 
labels, stock images and layouts are added to the images 
without further user intervention. 

Referring also to FIG. 8, the tour function also includes 
automatic annotation (150) of images obtained in each tour 
with predefined text or graphical information. The tour can 
also specify con^sition of printed or other output, com- 
bining acquired images and/or stodc images or diagrams that 
lend clarity and context to the images acquired (step 152). 
Page formatting allows multiple images in multiple sizes to 
appear on a single page or on multiple pages. 

Printed page layouts can be defined and modified based 
on a taxonomy defined when the user defines the functions 
of tour In one embodiment, the page description language is 
post-script level 2, but other interpreters for conversion to 
other file formats can be added. The tour mode script will 
define how images captured in the tour mode wiU be sized, 
scaled, displayed, arranged, formatted and labeled Images 
captured in the tour mode will define how they will be 
displayed, arranged and labeled. Additional drawings, 
diagrams, or stock images can be pulled into the defined 
page layout Once defined, the layout can be stored, or 
printed to hard copy or slides. 

Images captured randomly (outside of a defined tour) in 
an unstructured capture (step 146 in FIG. 7) are formatted by 
a user-defined algorithm, or a default format is used in the 
absence of user instructions to the contrary. This layout 
formats pages to conform to a predeteiroincd optimum 
numbo* of images and spatial arrangement based on the 
target printing or display device specified in the user pref- 
erence (as set forth in preference database 20). In addition, 
the user has the option to select the maximum nmnber of 
images to be displayed on a page on the selected device. The 
selection can be made at the time of printing. In the event no 
selection is made, the system will use the default format 
specified in the preference database. In all cases, image size, 
scale and position will be automatically calculated by the 
system prior to printing. 

Unstructured captures are printed in a manner such that 
they are visually pleasing and occupy the maximum imaging 
area available on the target output device. Hiese calculations 
are performed at the time of ou^ut, and are specific to the 
capability of the target printing device. Video printers are 
known that will print Va of a page if the 4 picture per page 
mode is set if only one image is in memory. In the present 
system, in contrast, the image print format is selected based 
on the number of prints to be printed, up to the maximum per 
page specified by the user preference. 

In a session in which both tour images and random images 
are acquired* a page layout for the prints will be generated 
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consisting of a tour format plus a random format based on 
desired output device. The user can configure and modiiy 
the layout at any time. If no user selection is made, a default 
layout is used. 

Referring to FIG. 9. system 10 also provides a simple 
technique for overlaying text or graphics on the acquired 
images (the technique can also be pcif ormed via a laptop or 
palmtop computer or other suitable computing device that is 
externally connected to personal con^)uter 12). The resultant 
image is a composite of the annotation and overlay^ so that 
tfic acquired image remains unchanged. The overlay text is 
anti-aliased for in^roved appearance. Anti-aliased text is 
used extensively throughout the interactions with the sys- 
teoL The anti-aUased t^ uses broadcast standard anti- 
aliasing technology in which text and background colors are 
man^ulated on a pixel by pixel basis to give the illusion of 
smooth transition and well-deiined text borders even while 
the system is <^erating in die RS170 display standards. 

Thus, the system formats acquired images to improve the 
communicating value of the images. Communicating value 
is enhanced by combination of the images with text infor- 
mation from the database or with user-directed text 
information, and by fcsmatting images into a printed array 
that may include backgrounds^ previously stored images^ 
text information^ and sizing, scaling, or orientation opera- 
tions. The formatting functions are automated based on 
user-defined or specialty-defined rules. Image formats arc 
selected based on context i.e., based on the script method 
described atxjve, the procedure, or the identity of the user. 

Text display by the system is performed in 2 modes: raster 
and outline. In raster anti-aliased text, a color map picture 
for each letter is used. Outline fonts are mathematical 
descriptions of the bounding lines of the text, and instruc- 
tions to fill the inner portion of the text 

The use of anti-aliasing technology is Loiportant so that 
the text looks good on the interlaced screen. 

The system provides the following important features of 
annotation: 

1. Tlie annotation does not damage the underlying image; 

2. The annotation can be positioned using either a 
keyboard, a mouse^ or other pointing device; 

3. The starting point for annotation box location can be 
specified; in one embodiment, the starting point is in 
the center of the screen, which minimizes the distance 
the user needs to drag the text; 

4. The color of the annotation (default is yellow with gray 
drop shadow) is selectable, but contrasts well with most 
endoscopic images; 

5. Rroportion and location of annotation is maintained 
when images are scaled for ou^ut 

Annotation generally starts at the image browser or 
Ubraiy page. The desired image is selected by positioning 
the cursor around the desired postage stamp size image. The 
image file is then opened. 

Starting with the image file 160, the operator (e.g., the 
physician) selects the text or graphical objects with which to 
annotate the image using one or more suitable data input 
devices 22 (FIGS. 1 and 2) (step 162). The input device is 
also used to locate the text or graphical object on the image 
(step 164). Object attributes such as font, color, and size can 
be selected manually or may be supplied by preference 
database 20 (step 166). The information that specifies the 
size, location, and other attributes of the graphical object arc 
stared as relative coordinates in an overlay (step 168). and 
thus do not affect the source image. Finally, the objects and 
text are resized as the image is scaled for display (step 169). 
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Printed output of the image/overlay composite will appear as 
if the overlay were part of the image itself. 

Annotation can be performed in a similar manner at both 
the image and page layout levels, A single image or page 
5 layout can possess multiple annotation overlays. Single or 
multiple overlays may be presented as a composite with the 
image. The overlay may also possess the attribute of 
translucency, ranging from obliteration of the underlying 
image to transparency in areas in whidi the overlay is not 
visible. Thus, the annotation can be partly transparent 
Single or multiple overlays may be attached to an image at 
any time. 

System 10 also displays system status and functions to the 
user in a format and style specifled by the user. A CPU- 
directed digital video kcyer 104 (FIG. 4) adds the status 

15 overlay to the video signal displayed by the system, in much 
the same manner as discussed above. The overlay includes 
text infomiation, graphics, or other images. Thus, the system 
can use software intelligence to display the status of con- 
nected devices or the stams of the system itself so that only 

20 important status information is displayed. The information is 
displayed in the way that Is most useful to the operator. 

FIG. 10 illustrates several different options available to 
the operator for entering the annotation information. An 
important functionality is **pen based" computing using a 

25 commercially available pen-based hardware 54 (FIG. 1) as 
the input device, but similar functionality may be obtained 
with other ii^ut devices. The endoscopic image is di^layed, 
and the input device (e.g., the pen-based system) is used to 
annotate, manipulate, or process the image as discussed 

30 above. Using the pen-based system, the operator actually 
prints or writes on the screen, and system 10 translates the 
pen strokes into machine-readable characters. System 10 
includes a search/shorthand capability to expand characters 
to an expeaed or defined word ca: most conunon "near hit." 

35 The pen-based system also provides an extremely 
straightforward device for allowing the user to draw Hnes, 
squares, and circles, and for controlling area of interest 
processing and color. 

As discussed above, annotations are stored as text and 

40 location so that when video formats are dianged, or images 
are scaled to larger or smaller sizes, the aspect ratio, 
location, and contrast of the annotations with respect to the 
images are maintained. 
As discussed in more detail below, system software 14 

45 (FIG. 1) includes an image processing module or engine for 
enhancing the visual quality of images. Techniques in^Ie- 
mcntcd by software 14 include multiple methods for video 
noise reduction, masking, smoothing, sharpening, contrast 
and histogram adjustment, contrast stretch and contraction. 

50 Software 14 also analyzes the image and adjusts die algo- 
rithms to optimize the image based on predictable 
procedure, specialty, surgeon, or other predictive factors. 
Software 14 also uses multiple analyses of the source image 
to dynamically adjust the image processing parameters. 

55 Image files created by system 10 are stinpcd with an 
internal "authentication" code not accessible to the user to 
designate the file as uncorrupted or structurally changed for 
legal purposes. Any image process that would substantially 
alta the information contained in the file causes this stamp 

60 (dirty bit) to be modified, thereby indicating that the file is 
no longer primary source documentation. The utilities pro- 
vided to cause image modification provide an opportunity to 
copy the image prior to modification so that alterations can 
be performed without compromising the evidentiary quality 

65 of the documentation. 

Referring to FIG. 11, as discussed above, video subsystem 
34^ processes images obtained during endoscopy for display 
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on virtually any type of display device and in any suitable In the infrared iinpleiaentation, the handheld infrared 
format lYcfcrcncc database 20 supplies the conversion transnutter 48 generates a signal that is handled In two 
information that specifies the file type of the output image possible manners: 1) the target device directly receives. 
andtheprocesstobeperfoniied(step 170). Video subsystem inteiprets and responds to the infrared emission of the 
34/ uses this information to process the raw image data 172 5 handheld device, or 2) the system infrared receiver interprets 
(step 174). It is immediately prior to this point that the ^hc infrared emission of the handheld unit, and then sends a 
authentication code is appHed to the image file (step 176). If command to another infrared emitter to emit an infrared 
desired, externalconversionutiUti^^^^ signal that can be interpreted by the target device. In the first 
me (step 177). The un^^ case, the handheld unit contains memory for codes to 
for display (step 179) many oftheformatslistedmFIG.il. „ j' ». * • j • 1 * . 1 
Processing ocirs via a prxJcess of color reduction, whereby ^'f''^ the correct infrmd signal, or sequence of signals, 
in the transition from a high (eg. 24) bit depth t6 a Iowa ^ '^^""^^ f ^V^^ ^^^'^ ^^"^^ ^ of the 
hit depth (e.g., 16, 8, or gray scale), the process is one of ^^8^^^' selection of signals. In this way, the 
calculated selection of specific color representations to be &y^^<^^ rcmams genenc, and able to support a wide vanety 
retained (as opposed to arbitrary file truncation). The color devices. In addition, usff-specific definitions can be 
reduction technique is based on commonly used algorithms 15 applied to the rules for selection of signals such tiiat the 
including frequency and domain calculation. The available same keypress may result in different operations of the target 
palette of colors is used to best represent the colors present device. Such preferences are stored in the preference data- 
in the source image. Processing occurs via a process of color base. 

reduction, whereby in the transition from a high (e.g. , 24) bit A provision is also made for support of a system in which 
to a lower bit depth (e.g., 16, 8 or gray scale), the process is 20 frame-by-frame address information is written onto the 
one of calculated selection of a specific color palette to best video tape. Frame addresses are stored in the imaging 
represent the colar fidelity of the source image, as o|^sed system database, and can be used to index, record, and 
to arbitrary file truncation. playback videotape on a frame-accurate basis. Frame- 
Referring again to FIG. 2, controls of a VCR 66 are accurate code is applied to the videotape by one of several 
programmable using one or more data input devices 22 (e.g., 25 commercially available personal computer interfaces. The 
remote control 48). Signals from remote control 48 are interface reports time codes to the host computer, while 
interpreted by a command interpreter in remote control simultaneously recording the frame information on a special 
adapter 34c according to prefaences stored in database 20 track on the video tape. The host computer relates the frame 
that associate actions taken by remote control 48, such as the numbering information to the database. Later, the host 
actuation of one or more buttons on the device, with a 30 computer retrieves the frame information, and issues a 
command. The command interpreter relays the appropriate command to the VCR control interface. TTie interface is 
command (e.g., for record, stop, pause and play) to VCR capable of controlling all functions of the VCR with frame- 
subsystem 34m, which in turn transnuts the commands to by-frame accuracy in response to commands issued by the 
VCR 66, either via a hardwired coimection or via interpreted host computer. A typical command interaction would be as 
and generated infra-red signals. 35 follows: The host issues a search command, and reports a 
In the hardwired inqslahcntation, the handheld infrared frame number retrieved from tiic database. The interface 
signal is processed by the receiver. A code is reported to causes the VCR to read the special track of the information, 
CPU 38 via bus 36, and CPU 38 dispatches a command from and advances the t^e to the desired frame, where the VCR 
the device driver specified in the database to the end device, pauses, waiting for the next command. A "ready" response 
in this case, a VCR. The command is dispatched via an 40 is returned to the CPU via the interface, and the next 
RS232 interface, or a device-specific interface. In the RS232 command is issued, 

in^lementation, the device may return an acknowledgment System 10 is also capable of interactive recording or 

or error response that is interpreted and processed by the playback of audio signals. The audio material is written to 

CPU. In all cases, the device driver contains the logic and and played back from digital files by an audio ad^ter 

rules for interpretation, such that the command processor 45 subsystem 34 (not separately shown in FIG. 2). 

and CPU interact with controlled and control devices in a Referring also to FIG. 12, the operation of system 10 in 

consistent and standard manner. response to the remote control devices that can be used with 

The invention si^ports the use of an internally mounted the system is shown. As discussed, the remote control 

super-8 format S-video recording apparatus. This apparatus devices include infrared remote control 48 and voice pickup 

is supported as follows: The system utilizes high quality 50 50. Other remote control techniques, such as detection of 

vidco-8 and hi^-8 standard mimature video tapes. The pupillary eye movement or the closure of contacts on, e.jg., 

operation of the system (record, play, tape motion and a footswitdi, may also be used. 

power) are controlled by an interface to the CPU, as a Voice pickup 50 enables the operator to perform spoken 

subsystem. Tape counter information related to the relative actions. The spoken speech is processed by a voice activa- 

position of the tape is communicated to the system database. 55 tion subsystem. The speech input is compared to the pref- 

User commands are issued to the device using any of the erence database taxonomy, and the appropriate action is 

system-supported control interfaces described hereinafter. communicated to the target device impacted by the com- 

When a user issues a record command; the system actuates mand. 

the video recorder for a specified number of seconds. The Thus, actions taken by the operator can include speaking, 

system indicates the record operation to the user via a screen 60 pressing a button, contact closure, or other actions. 

GST audio signal. An icon representing the video clip is saved In response to an action taken by the operator, such as the 

in the image browser. An audio annotation file describing the depression of a button on IR remote control 48 (step 180), 

video dip or a text description of the infarmation in the the subsystem 34 associated with the device (in this 

video dip can be associated with the icon. The icon itself exan^3le, remote control adapter 34c) detects the signal (step 

contains the tape counter address of the clip. The system can 65 182) and interprets the signal (step 184) using information 

issue commands to the tape device to queue the device to stored in an internal memory (not shown). Subsystem 34c 

play the designated segment of the video tape. then accesses preference database 20 (step 186) to determine 
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the action to take in response to the operator's command. by a given host device, the host machine interface interprets 

The action is communicated to the device driver for the the user preference and substitutes the closest available 

device that is impacted by the command (step 188), and the option that the host unit is capable of i^oviding, or desig- 

action is performed. Thus, a generic action taken by tiie nates the preference selection as unavailable, 

operator is translated to a device-specific instruction accord- 5 Qeation of a new user requires specification of user login 

ing to the preferences of the particular operator stored in ID, password, secondary password, and specialty. Default 

preference database 20. configurations that can include video parameters, forms, and 

The same action can result in different conunands or predefined image capture sequences are selected based on 

actions depending on when in the procedure the action is the predicted needs of the operator based on specialty, 

taken, or what procedure is in effect at the time. The lO Operator-specific modifications can be performed at the time 

taxonomy of responses and logic for response selection is the database is created, or at a later time by editing the user 

stored in the preference database. The commands are issued preference database using tools provided by the user inta- 

and received as generic commands, and then interpreted by face for that purpose. The function of logging on to the 

the software of the system based on the rules spedfled by the system, once a user is created, provides password security to 

preference database. The responses are governed by identity 15 user data and results in loading of user-specific setup param- 

of the user, the procedure, or other information input into the eters. A secondary password is provided to allow other 

system. Additional functions can be added, using a minimal personnel, such as nurses, access to preference and patient 

number of controls at any given time. Because the selections data records with a restricted set of permissions defined by 

will be relevant to the context of the job at hand, the the primary account holder. 

functions presented to the user will be the smallest sub-set 20 Each removable media stands alone, and contains all 

of ail system functions required to complete the given task information needed to initiaKze and load preferences of a 

at hand. In addition, the actual operation of the functions can user on any host system. When removable media is filled to 

be adjusted such that the increments of the response will be capacity, the system provides a means to transfer user data 

tailored to the expected range of operation in a given to prepare new media. The system is independent of the 

circumstance. For example, in a procedure in which large 25 identity of media, or specific media type. The only require- 

differences in brightness are anticqjated, the incremental ment for using media on the system is that the system 

response to a single brightness increase command is greater contain the required hardware drive to support the specified 

than the incremental response to the same brightness media. No sequencing or labeling of media js required. As 

inacase command issued during a procedure in which such, the host unit can read information entered by any 

smaller differences in brightoess are anticipated. Further, 30 system, as long as it is stOTcd in the ap|H-opriatc format on 

within a single operation, the sensitivity of increase is the disk. Similarly, data firom the removable cartridges can 

dynamically, or ballistically, compensated. Issuance of mul- also be utilized by database programs external to the sys- 

tiple repetitive commands results in increasing incremental tern's database. 

change. In this way the systemis both rapid in response, and In one embodiment, the video subsystem provides the 
sufficiently accurate in response. 35 ability to optimize and enhance both live and still video 

The system integrates and distributes commands to the image display based on available system resources and user 

appropriate subsystem. Le., one or many mechanisms to preferences stcffed in the database. The video subsystem 

capture operator desires can communicate to one or more controls are executed by software conunands issued to the 

subsystems. video subsystem by a subsystem-specific software driver, 
The available input devices can include keyboard, mjause, 40 under control of the main software engine. The functions of 

hardwired and wireless remote control, speech recognition, this portion of the invention relate to decoding and digiti- 

and handwriting "pen based" devices. The input devices zation of the incoming signal, processing the signal, acquir- 

accept generic commands such as keypress, switch closure, ing digital image files and presenting them to the host 

or receipt of an infra-red signal. The input handling engine interface, digital to analog conversion, and encoding of 
of the interface compares the generic input of the control 45 output video. Each step of the process is under software 

device, matdies it to the action specified by the user pref- control mediated by device-specific drivers that implement 

erence database (described below) and issues the correct the conunands by sending control commands to the graphic 

command to perform the action or response specified in the processor, A/D and D/A converters and encoding or decod- 

prefcrence database. In this way, the same generic action can ing chips, arithmetic logic units, or digital signal processor, 
result in issuance of different conunands based on the 50 Incoming video signals are decoded to RGB as required. In 

context of use, identity, or other characteristics of the uset the RGB domain, fiinctions arc performed relating to inaii- 

The database is structured to contain system-specific aging the video input and output devices induding adjust- 

information, user prcfaenccs, patient text and image data. ment of pixel dock rate, synchronization of video signals 

The user interface provides different and function-specific from multiple input sources, conversion of a variety of input 
access to each of these database functions. System-specific 55 signals to different output signals or resolutions, histogram 

information relates to available system resources and jh-^- operations on an image, contrast expansions, range and 

ferred configuration. This information is stored on the main offset adjustments, ovCTlay functions, combinations of video 

system hard drive. Access to system-specific information is inputs, alternation between frame buffer storage of a given 

provided in a structured menu accessed by entering "setup" frame and continuous viewing of Uve video, magnification, 
at the login screen. Senip is a reserved login ID, outside the 60 scroU and pan functions, look up table conversions, frame 

normal end user operation of the system. averaging, frame doubling, and other functions. These fiinc- 

User preferences and identity, in contrast are stored on tions are performed by a combination of a TMS 34010 

removable media, and are accessible only after creating or gr^hic processor, an arithmetic logic unit, digital signal 

logging into a user account by entry of an appropriate processor, analog-to-digital converters, digital-to-analog 
password. In this way, user preferences are specific to the 65 converters, or encoding or decoding chips as required. At 

user and portable between different locations or host output, the RGB signal then undergoes encoding required 

machines. In die event that a user preference is not supported for the designated display device. 
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EDhancemeat of Still Images: 

Stored images are enhanced by a user selection option for 
enhaacement that implements a series of standard median, 
convolution, and contrast enhancement algorithms. 
Enhancement algorithms are run as monochrome algorithms 
on each of the red, green, and blue planes of the image where 
each pixel consists of an x,y coordinate and a value fco^ the 
pixel at the coordinate. In one implementation, the color 
value is specified as an 8-bit value, although any value can 
be used. The routines to handle this are implemented by 
Signal Processing engine, (IGGY 360, described below). 
The first step in processing is to copy the image data into 
EMM. Image processing is performed in memory for speed. 
The original image file name is changed^ so that the original 
image can be restored after the processing functions are 
completed. The algorithms are written as resolution Inde- 
pendent Resolution changes are handled by bi-directional 
interpolation. linear interpolation can be useid for speed, but 
bicubic interpolation is useful for quality. The preference 
database contains the exact sequence to be followed for the 
given instance of selecting the enhance features. The pref- 
erence database also contains the information regarding the 
desired input and output resolutions, and parameters for 
each processing algoritiun (eg., sharpening factor fcH* sharp- 
' cning algorithm). These values arc passed to IGGY 360 at 
the time the enhance selection is made. 

The following example assumes a pixel value of 8 bits per 
red, green and blue value, for 24 bit color scheme. Hie 
image matrix is first interpolated to double the spatial 
resolution of die source image using bicubic interpolator 
801. Next an auto crop function is performed. The maximum 
gray level (corresponding to black for the image) is traced 
radiaUy outward from the central point of the image matrix 
until 5 consecutive pixels reflect an assigned maximum gray 
level (conesponding to black fw the image). In this way the 
border around an image (which occurs, e.g., when one is 
using an arthroscope) is cropped from the image. If the 
border criteria is poorly defined (i.e., the edge is not 
detected) a bounding band for the crop is presented to the 
user for verification or modification using a pointing device. 
Image data outside the crop boundary is not modified, but is 
excluded from image distribution calculation and subse- 
quent processing steps. Next, a distribution of the pixel 
values versus count is computed for each of the red, green, 
and blue con^nent data matrices. In most cases, only a 
portion of the available range of 256 possible values is used. 
The values for the pixel values are remapped so that lowest 
pixel value is assigned the value of zero, and the highest 
pixel value 255, and each intervening pixel value distributed 
among the remaining values by multiplying the original 
pixel value by a ratio of V2-V W2. where V2 is the total 
range of the new distribution (Vmax-Vmin) and VI is the 
total range of the original distribution (Vmax-Vmin). In this 
way the original pixel values are distributed across the 
maximum range possible for the system while preserving the 
same relative proportions of each pixel value to the total. 
Next an appropriate shaipen convolution or median filter is 
applied to the data using the shaipen <x filter parameter 
specified by the preference database. Shaipen and median 
(noise reduction) filters are standard algorithms based on a 
three pixel square (9 pixel) matrix used on each of the red 
green and blue monochrome images. In the sharpen filter, 
the values for the upper left and upper central pixels are 
multiplied by 50% of the value of the sharpen factor. The 
upper right corner pixel is mulUplicd by 100% of the value 
of the sharpen factor. The lower left pixel is multiplied by 
-100% of the sharpen factor, and the lower central and lower 
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right pixels are multiplied by -50% of the sharpen factor. 
The left and right central pixels are not changed. The 
resultant value for all 9 pixels is summed, divided by 9, and 
the central pixel of the matrix is replaced with the resulting 
5 value. The process continues until all pixels are updated. For 
noise reduction, a similar matrix of 3 by 3 pixels (9 pixels) 
is used, replacing the central pixel of the tile with the median 
value, or the median value multiplied by a factor stored in 
the database. 

10 The interlaced noise reduction filter reduces the motion 
artifact induced by interlaced acquisition of video. This fUta 
reduces the "hounds tooth** appearance that results when 
motion occurs in the source image between the acquisition 
of the odd and even fields of video that comprise the frame. 

15 This filter results in superior correction of the artifact 
without the resolution loss inherent in frame doubling or 
frame averaging. The filter works by comparing mono- 
chrome pixels in the red green and blue planes. Pixels in 
scan line n and (n-f2) are compared to the pixel value of scan 

20 line (n+1), starting with scan line 1. If the value of pixel in 
scan line (iH-1) is more than (a) units (where (a) is a value 
specified in the preference database, specified as a=^ in one 
implementation) greater than the value of ((n)+(n+2))/2, 
(where (n) and (n+1) are the values of the associated pixel) 

23 then pixel (n-f 1) is replaced by the value of ((n>f(irt-2))/2. If 
the difference between the calculated values is less than or 
equal to (a), then no action is taken. The result is that when 
significant differences exist between even and odd fields that 
comprise a frame of video, the difference will be minimized. 

30 When the difference is within the factor specified no data 
will be altered. Again, this algorithm is implemented on a 
data matrix that has been doubled by using bicubic interpo- 
lation. 

At completion of the specified chain of processing, the 
35 interpolator is called to scale the image to the desired display 
size. At completion of the processing, the image is trans- 
ferred from memory to the display device or the disk 
memoiy. 
Video Printer: 

40 A video printer is operated as follows. The system gen- 
erates a video signal that serves as the input for the video 
printer. The video signal is created by the system, as opposed 
to the live video signal produced by the video input source 
during surgery. The signal produced by the system can 

45 include image enhancement, annotation, and printing of 
multiple images on a single print. 

The system generates control signals and receives control 
signals from the video printer via a serial connection. These 
signals can be used to replace the operator input normally 

SO required to generate multiple copies of multiple images. 
Support of Digital Printing Devices: 

Mntcars capable of imaging and control via digital means 
can be controlled by serial, parallel, SCSI, GPIB, or other 
interfaces. In fiiis instance, the system configures a file that 

55 consists of all raster data sufficient to produce the image, and 
device-specific digital commands to control the printing 
process. This information is passed to the printing device, 
and status is reported to the system via the interface con- 
nection. In this implementation, the invention performs all 

60 scaling, raster image processing, and colc»r operations 
required to drive a printer that supports only raster data- 
Post Script Compatible Devices: 

Printers or devices containing raster image processors 
compatible with post script are handled as follows: The page 

65 description, control commands, and raw image data are 
downloaded to the printer via one of the above-described 
interfaces. The actual conversion of page description to 
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raster data, including scaling, color corrcctioD. and other handled byretumtodisplay of live video mode, with a small 
processes arc handled by the raster image processor con- annotation of the error at the top of the screen. Fuithcr 
taincd in the device. This configuration is much faster than operation of other options will be restricted until the con- 
thc transmission of raster data to the connected device. dition causing the citot is corrected. 

Interface Operation: Heuristic Process: 5 Refening to FIG. 14, the device handling architecture 192 

The system minimizes the number of options presented to implemented by system 10 is illustrated. This architecture 
the user at any given time. The system will select and present allows many devices 16, 18, 22, 26 (FIG. 1) to be supported 
to the user the smallest subset of available system resources by a single operating program 14. Custom configurations are 
that is likely to be used for the job at hand. The selection stored in a taxonomy in preference database 20, which is 
process is based upon: 1) the resources currently available to lO referenced during the configuration and operation of system 
the system; 2) the resources likely to be used at a given time; 10. This architecture, and the structure and organization of 
3) identity of the user, or the procedure; and 4) the risk of the database (described bdow) allows system 10 to be 
inadvertent operation of the system during critical functions. platform and device independent — that is, when a device is 
The preference database contains the rules for operation of added to or removed from the system, only the software 
resource identification and allocation, so that the rules may 15 driver that controls that device need be changed. Portions of 
be updated as system usage dianges, or as new requirements the programs or device drivers may reside in host processor 
or devices are added. 30, the device itsdf, or on other media or memory accessible 

Available Resources: to host processor 30 and the device. 

The system will display only those options that pertain to FIG. 15 illustrates the connectivity and operation of a 
features that arc actually installed and operational. At system 20 PCMCIA adapter 196 (e.g., adapters 34/-34i and 34p of 
startup, and intermittently during operation of the system, a FIG. 2). PCMCIA is an evolving standard for an array of 
poll of available devices will be pcrfOTmcd. Devices that are devices that include storage^ conununicating and other 
present and operational will be made available to the inter- external functions. In syste m 10, the PCMCIA (or any other 
face. Corresponding icons will display based on the likeli- suitable standard, such as JEOA) is used to interface to 
hood that they will be used. Systems that respond with an 25 multiple input and ou^ut devices in a standard mannex. 
error message will activate device-specific diagnostic rou- PCMCIA ad^ter 196 is self-contained and durable, and is 
tines to attempt to diange the device to an operational mode, equipped with a standard edge connector for ease of use. No 
Off will activate device-specific error handling messages. additional wiring is required; the adapter slot provides 
These error messages are ported to the user interface by operating power. Thus, the action of inserting adapter card 
changing the color of the affected icon to red, and by issuing 30 196 into the slot in host processor 30 is all tfiat is required 
a text error message that reports the identified fault. Issuance to initiate appropriate identification and nianagemcnt of the 
of the error repeat is prioritized based on the likelihood that device that is insetted. 

the error will impede operation of the basic endoscopic A PCMCIA adapter 196 may alternatively be located 
functionality. For example* deficits in a video acquisition externally to host processor 30. For example, a PCMCIA 
source will immediately flag the operator, and suspend 35 adapter 196 may be disposed in another device (such as a 
further operation of the systena. Non-critical deficits, for laptop or palmtop computer 54 (FIG. 2) that communicates 
example, a printer for which paper has run out, will flag the with host processor 30). 

user when the device is selected. FIG. 16 illustrates options 200 provided to the user by 

Context of Operation: preference database 20 and shows how the user and CPU 38 

Only available options that are likely to be useful in the 40 interact with preference database 20. Options 200 include 
context of the current operational mode of the system wiU be specifications as to video format 202, processing 204, the 
presented at a given time. For example, during live video video sources to be used 206, and the configuration of the 
operation of the system, live video manipulation and image hardware 208. In addition, information concerning the users 
capture operations will be available. Options not likely to be 210 and the hospital 212 (or other facility in which system 
used, such as printing or output selections, or options likely 45 10 resides) is also provided. The attributes of the stored data 
to suspend operation of live video, such as image 214, relationships between the data 216 (e.g., one-to-one, 
processing, are not presented to the user. one-to-many, niany-to-one, and many-to-many) are also 

Identity of the User; provided, as are calculated data field attributes 218, such as 

Based on the user preference database, the user is pre- required entry, text, date, numeric, or dedmaL Other types 
sented with a user-defined subset of devices and functions. 50 of data fields are date and time. 

The available functions are a subset of aU available fimc- Different physicians who use system 10 typically have 
tions and operations. different preferences as to how endoscopy images are to be 

Two user sign-on passwords are provided. The preference obtained, processed, stOTed, and displayed. Indeed, a given 
database contains a table of authorities that determine which physician's preferences for tiiese options may vary accord- 
functions a user may access. Based on user training, and 55 ing to the procedure being performed, or even according to 
authorization, specific functions of the system may be the particular stage of any one procedure. Preference data- 
restricted. For example, inexperienced users will have base 20 classifies the preference infonnation according to 
access to system initialization, or preparations for cndo- the individual physician, specialty, procedure, and stage of 
scopic operation, but wiU be denied access to potentially procedure to give the physician the capabiUty of configuring 
destructive operations such as image or record deletion. 60 and operating the devices in system 10 rapidly, according to 
Likelihood of Inadvertent Operation: his or her pre-stored preferences. 

During Uve video operation, the user is presented with the When the physician begins the procedure, the physician 
most restricted subset of available features and operations. enters his or her name, specialty, and any other information 
Most of the features presented relate to the operation of the that corresponds to the physician's desired setup, e.g., 
real time image processing features. All features have an 65 identification of the procedure to be performed (step 220). 
immediate escape, or return to live video operation. Errors CPU 38 retrieves the preference information for this physi- 
that occur in an option selected from the live video mode are cian and procedure and uses it to configure the devices (e.g.. 
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cameras, CRT displays) prcfoentially used by the physician 
during the procedure (step 222). CPU 38 also maiotaios and 
updates the preference information in rcspoose to changes 
ordered by ^e physicians, 

HGS. 17 and 18 illustrate procedures for storing images 
in and retrieving images from an image database. The image 
database can reside in any suitable memory device in system 
10 (e.g., disk storage). 
Slide Presentation System: 

The system automates production of slides by means of a 
style engine. The style engine takes simple text inf onnation 
to be displayed on the slide, and produces a complex 
graphical design that is pleasing to the eye, and consistent 
throughout a presentation. The graphical design includes 
color^ spatial arrangement, bullet marks, justification, and 
other graphical enhancement instructions. The style engine 
automatically positions text, graphical elements, and images 
without additional operator input, other than the user typing 
in the desired text or selecting the desired image. In one 
embodiment the styles are tailored to meet the needs of the 
medical presenter, but the engine is generic, and additional 
style parameters can be added. 

The style engine is also responsible for updating existing 
presentations. Selection of a style changes the appearance 
and attributes of slides previously created, or imported. 
CqIots, sizes, positioning, font, and text selections are 
changed so that all images in the presentation are consistent 
in look, fed and layout 

Following are some examples of the style engine func- 
tionality: The slide construction screen has designated areas 
for titles, subtittcs, body text, bullet text, multi-column text, 
and images. Not all slide screens contain all elements. The 
user simply enten text, and the style engine adjusts the font 
size, i>osition, and color of the ^de con^sition by style 
engine rules. The style engine determines what information 
the user has entered, and maximizes use of the available 
imaging of the slide film. Elements are sized and colored 
according to a schema for ^e designated style. Each element 
is dynamically scaled so that the proportion of title to 
subtitle to body text or other elements is maintained, and the 
margins are not violated. The engine automatically compen- 
sates for elements without entry by centering, sizing, and 
scaling the information on the slide. Algorithms are used to 
detemune leading spaces, whidi are preserved, from trailing 
space, which is truncated. When proportional fonts are 
selected, the style engine performs kerning operations to 
normalize the appearance and spacing of the text The style 
engine win deteimine global limits from text font and size 
for a given fsesentatioa, so that slide to slide variation of 
font, size color, and positioning will be constrained accord- 
ing to good presentation guidelines. These limits, or 
guidelines, arc stored in text lUcs, so that the operating range 
of the style engine can be modified at any time. 
Clipboard Function: 

Images and data can be collected from across many 
records or storage devices without respect to user or media 
identity using the clipboard device. A record or device is 
designated as the recipient for data. Searches of single or 
multiple databases and devices may be performed without 
respect to media or database identification. When the desired 
information is located, the clipboard function copies the 
selected information or images to the specified recipient 
media and file. 

The clipt>oard mechanism can be used to collect images 
and data from across multiple users, multiple media, and 
multiple storage formats or databases. This function is used 
to collect a pertinent subset of images, across media, formats 
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and users. For example, a collection of images may be 
CTeated to provide ready access to images to be used by the 
slide module in creating a presentation. This mechanism can 
also be used to iiiq)ort or export data to external database 
5 applications or storage devices. For exan^jle, a small num- 
ber of relevant images may be placed onto a PCMCIA 
storage device for use in a laptop computer or presentation 
program 

Arithmetic Logic Unit: 
10 The system includes signal processors running at real- 
time pixel clock speeds that perform signal-to-noise 
analysis, and apply correction to the signal prior to display. 
Slide-Making F\inction 
Referring to FIG. 13, system 10 also performs a slide- 
15 making function 190. Slide function 190 allows: 

1. rapid and easy integration of high quality digitally 
processed images acquired using the OR image capture 
system discussed above; and 

2. a method for constructing presentations, A presentation 
consists of a series of slides that contain formatted 
layouts of image, text and graphic infonoation. 

Image information may be used by many presentations, 
and stored only once. The system database (discussed 
below) allows for this by supporting "one-to-many" and 
"many-to-many^ data relationships. Slides, images on 
slides, or any other element con^irising a slide may be 
shared by greater than one presentation. An object wiU 
continue to exist until all presentations that call it are 
modified ox deleted. 

Slide function 190 performs the following operations: 
Image Processing Functions 

This part of the system allows the user the ability to 
specify parameters normally specified by the preference 
database, thereby allowing manual conurol of digital image 
processing functions in addition to the preference database 
based processing methods. 
Scaling and Positioning 
Multiple images may be scaled and positioned on a single 
^ slide. The appropriate scaling and positioning are done on 
the fly from the appropriate source image, leaving the source 
image unchanged. 
Text 

High quality anti-aJiased text is available in multiple fonts 
45 and point sized for overiay and text functions. 
Preview Mode 

Image information is displayed in a '^ugh cut" or 
approximate view version for processing speed. A presen- 
tation may be viewed as multiple miniature representations 
50 of the con^nent slides on a single screen. The slides can be 
moved or modified using any pointing device to restructure 
the presentation. This can be done on a laptop, personal 
computer, pen based system, or workstation. 
Batch Print 

55 A presentation may be queued for printing. The system 
will automatically control a digital film recording device, 
tninsfaring the digital files to film. Automatic recording of 
number of available frame and errors will be incorporated. 
This should allow for most slides to be printed and 
60 discarded, allowing all filing to be conveniently located on 
the machine. 
Preview and Edit 

A simple interface allows the operator to view and edit 
slides in a so-called '"what you see is what you get** 
65 presentation. During the preview mode, the aaual visual is 
formatted and presented to the user at the full resolution of 
which the selected display device is capable. 
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C) Menu Background Graphics 
a) RA Graphic token list 

D) Menu Help Data 

E) Menu EMM memory address 
5 F) Menu efficiency tables 

G) Menu Button InformatioD 

a) Button Count 

b) Button ID 

c) Button Location 
10 d) Button Size 

e) Button Grajf^cs 
1) RA Graphic Token List 

f) Button Function Pointer 

g) Button Attributes 
15 h) Button help text 

^th this technique, the entire control path of the appli- 
cation is described in data airays, instead of embodied in 
code. This flexibility allows the program control, as weU as 
the look and feel of the program, to be changed by simple 
20 manipulation of data, instead of with code. Furthermore, this 
technique allows the program itself to make critical control 
path and looktfecl changes, on the fly, while the user is 
running it. This capability allows the users to custom design 
the program to their tastes, affording an individualized fit to 
25 each and every user. The only thing the applications pro- 
grammer needs to |H:ovide is the controlling data arrays 
(described above) and the functionality (action) for each 
"button" described in those arrays. 

MOM 300 is designed to handle any number of "win- 
30 dows" on a display screen. The above data arrays describe 
both the look and feel, as well as the functionality of each 
window. A '^window" may consist of a background and one 
or more buttons. A background describes how the window 
appears, and is not limited to any particular look. A button 
35 is an entity that exists on top of the background^ and 
provides either information or functionality. When provid- 
ing information, the button retrieves some data and displays 
it to the user in an understandable manner. When providing 
functionality, the button causes some action or actions to 
40 occur when the user "pushes** the button (e.g., by touching 
the button on a touchscreen or activating it with a mouse). 

MOM 300 uses an integrated help system to allow the 
user to query both windows as well as buttons to see their 
functionality or use before being required to actually use the 
45 system. As can be seen by the above array, the help system 
is embodied in the menu and button descriptions themselves, 
and thus afford easy change and upgrade with the software. 

MOM 300 allows the display of windows to overlap* and 
handles complete maintenance as to the order and depth of 
50 windows, as well as the displaying of the windows that 
become revealed through use. 

In addition to the multi-window approach used by the 
menu manager, MOM 300 also provides multi-processing 
on the window level, as the user is never restricted to 
FIG. 20 shows the interconncctivity of the engines, and 55 working only on the *'top'* window. Through dynamic array 



System Software — Overview 

The external view of system 10, to the end user, is that of 
a medical-specific application, designed to handle both the 
capture and storage of real-time data, as well as the main- 
tenance and manipulation of the data and relationships of the 
data to other associated infoimatioD. This external view can 
be represented by FIG. 19. 

As shown by FIG. 19, the end user first encounters a login 
procedure 240, which admits him or her to system 10. The 
user may then go to either Data Acquisition (Procedure) 260 
or Information Processing (Query) 250. Both of these two 
sub-systems can invoke a signal processing subsystem 270 
and an output subsystem 2S0. 

The (simary non-medical features of program 14 (FIG. 1), 
from the end user's view, include: 

1) Simplicity of use 

Program 14 has been designed to be as simple as possible 
to the end user. This means that program 14 uses heuristic 
j^>proaches, in many cases, to avoid having to ask complex 
questions to the user. 

2) Platform Independence 

I^ogram 14 is designed to run on a multitude of platforms, 
and to allow many forms of input and output. This allows 
program 14 to be easily maintained, upgraded and expanded, 
and is not limited to availability of specialized equipment 
Itogram 14 is designed to run on multiple platforms, includ- 
ing DOS, WINDOWS, UNIX, NEXTstep, and workstation 
architecture. 

System Software — ^Technical Description 

Program 14, as illustrated in FIG. 19, embodies a much 
larger set of engines (i.e., software modules) running under 
the surface. Each engine is designed to provide a fairly easy 
mechanism for maintaining a flexible environment Flex- 
ibility includes the ability to make major changes in both the 
functionality and presentation of the software, as well as to 
provide platform independence for easy integration of new 
software and hardware. 

Refcffing also to FIG. 20, program 14 is divided into 
several engines each of which is devoted to a particular 
portion of the application, and is designed to integrate well 
with the other engines in the application. The engines 
included in this application arc described below; 



MOM 300 
EMM 310 
RA320 
DAD 330 
THE 340 
CSNERAL350 

IGGYSeO 
POP 370 
SIS 380 



Menu taanafement ei^ine 
Meamy naiiagcmcnt cnjinc 
Raster engine 

Platfom iulepcndent display engine 

Input deivice engine 

general haaXiers for system 

iadepcodcDce 

Sigml processing engine 

Output device eng^ 

Database enjpne 



their overall location within the application. 

The following sections describe in detail each of the 
engines, its purpose and features: 
Menu Manager Engine (MOM) 300 

Menu HLanagement engine 300 handles the creation and 
maintenance of the entire software application that executes 
in system 10. To control MOM 300, the application 
describes menus as a series of encapsulated data arrays. The 
data airays embody the following information: 

1) Top Level Menu Array 

A) Menu Location 

B) Menu Size 



65 



switdiing, MOM 300 allows the user to select any visible 
button, regardless of the associated window or its depth in 
the screen. Because of this, MOM 300 in^lcmcnts a flat 
window scheme in which no window has precedence over 
any other window. 

By providing this level of management, the ease of use to 
users is grcatiy improved by always allowing the users to 
"have their way." However, to provide total flexibility, 
MOM 300 does include an ability to simulate the more 
traditional hierarchical menu structures, if the need arises. 

MOM 300 performs its work in a fairly straightforward 
manner. When an application begins to run the first thing 
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done by the program is to invoke MOM 300 routines to open RA 320 is driven directly by MOM 3 00, and in some cases 

one or more windows. Once this is accomplished, the by the application functionadity itself. RA 320 is a table 

program then invokes MOM 300, which takes complete dispatched service and takes as input a pointer to a hetero- 

control of .the program from that point on. geneous data array. This data array consists of pointers and 

MOM 300 analyzes the existing menu structures and 5 data, where the pointers correspond to a token list that 

begins to query the users requests via THE engine 340. embodies a primitive graphics language. 

Display of the menus, as well as display of the user's input The token list described by RA320 contains the minimum 

device and sdcction of choices, is handled within MOM 300 needed set of drawing devices to render all menu drawing, 

by calling RA engine 320 to draw the desired information, This includes gradient rectangles, anti-aliased and non-anti- 

and DAD engine 330 to display it to the current output aliased text, circles and lines, as wcU as more complex 

device. concepts such as buttons, sliders, controls, LED's, and 

No rules are placed on what can be performed by the textured backgrounds. In order to provide capabilities that 

functionality invoked by MOM 300 via the data structures. may be needed later in the development cycle, RA 320 also 

Ifdesired, windows that were opened at the beginning of the provides programmer definable tokens, via a functional 

program may be dosed, and new windows opened. token, with which programmers can vector to their own code 

Likewise, any number of additional windows may be during a RA token execution. This enables the program to 

opened. These actions may occur in response to users draw any type of object, even objects never conceived 

activating various functions on the existing windows. This before in the software. 

may even include the restructuring of buttons and data RA 320 is limited to drawing to a single window at a time, 

within existing windows. For exarr^le, the user may select and may not mix drawing to different windows within a 

a button that causes half of the buttons on a current window 20 single token download. Therefore, it is up to MOM 300 to 

to be replaced with an image. This type of dynamic remod- direct RA 320 as to the appropriate window for drawing, as 

ding ofthe menu manager envirormient is totelly handled by well as providing RA 320 with the correct token list, as 

MOM 300. The functionality merely expffesses the changed extracted from the menu arrays. 

data arrays and MOM 300 handles the reintegration of the Because of the self-modifying nature of the software, as 

data into the existing environment in a seamless way. 23 well as its platform independence, RA 320 is designed to 

MOM 300 is also responsible for aU event handling at the ^Uow self-modification during the execution of an RA token 

menu level. MOM 300 can be made to poU events at every ^ ^^^^ self-modify while 

input device ^cle and maintains among other thmgs, a executing, aUowing decisions to be made on the fly, as to 

leal-tune clock for toing of events ^^d maintenaii^^^^^ a ^ down-stream from the current 

tune-of-day dock. MOM 300 uses THE engme 340 for - t f * v 

event polling, but allows button functionaUty to uncouple ^ ^^'^l^Vn 1 . - -,in • . a a 

event ^lUng for specialized event handling within a but- 'V^f ^"""^^ "^^^.^T"^,^^^''^ a standard 

ton's ^on. method. While the method is modifiable, it is a fixed in code 

MOM 300 never terminates, unless -aU windows are programmer. For example, within this application, 

dosed without a new window being opened prio* to func- 320 is defined to be 24 bit (8 bit per pixd) in a Packed 

tionality return. Besides this method, the only other method 35 architecture (RGBRGBRGB . . . ). By slotting RA 320 and 

of tcrmiiuition is direct exit within a function. DAD 330 apart, and allowing DAD 330 to handle device 

Mem«y Management Engine (EMM) 310 particulars, DAD 330 can color reduce/expand, as well as 

EMM engine 3 10 is responsible for the allocation, main- resolution reduce/expand the RA maintained EMM memory 

tenance and freeing of the data arrays used by RA engine buffers when displaying to any device. 

320 and the other functionality portions of ^e program. 40 Font drawing is handled inRA320 by use of precomputed 

Despite (he fact that modem programming software pro- raster font tables. The font tables contain one byte per pixel, 

vidcs memory allocation, the use of EMM engine 310 which indicates the natural antialiasing of the font at that 

assures platform independence when moving between soft- pixel Fonts are encoded so as to be both proportional as well 

ware platforms as well as hardware memory platforms. as non-proportional, and any font may t>e displayed in any 

EMM engiae 3 10 is a fairly simple code system that 43 combination of antialiasing and proportionality. When fonts 

allocates memory in '*pages" or '^blocks" when requested by are displayed, an additional font transparency sets the levd 

the rest of the software^ EMM 3 10 maintains data Lists to of opadty, allowing for drop shadows and other effects. Font 

track the memory, and provides boundary chockuig as weU color is specified as a 24 bU (RGB) value, whidi is used with 

as fredog of the data when its use is complete. the transparency and antialiasing information to calculate 

Finally, EMM 310 also provides the ability to query the 50 the appropriate color for each pixd. The software allows any 

memory system for remaining quantity and availability, number of fonts to be loaded concurrently, and fonts may be 

allow the software to provide status, as weU as predict switdied dynamically, on a character by character basis if 

remaining capability. desired. Font concurrence is only limited by available 

EMM 310 is used by MOM engine 300 to allocate the memory (as all fonts used are always resident), 

memory used for the windows (see the section below on RA 55 Platform Independent Display Engine (DAD) 330 

engine 320 for a more detailed description) as well as being DAD engine 330 handles actual updates of information to 

available to all the other engines as needed. the user's end display device. Since the software system is 

Raster Engine (RA) 320 device independent. DAD 330 coordinates requests from 

RA engine 320 is the main drawing engine responsible for MOM 300 and the user's functionality, and reads EMM 

drawing the menus and their assodated buttons and back- 60 memory and displays the contents of EMM memory to the 

grounds. Because this software system is designed to be current device. 

platfoim independent, RA 320 does not draw to any output Basically the cyde works as follows: MOM 300, follow- 

device. Instead. RA320 interfaces with EMM engine 310 ing advice from the menu structures, instructs RA 320 to 

and draws to EMM allocated memory. Actual display of the draw the various windows to EMM memory. After RA 320 

windows on the users display device is handled by DAD 65 completes the drawing, MOM 300 then instructs DAD 330 

engine 330. This type of architecture is termed a virtual to take the completed EMM memory and display it on the 

frame buffer. current display device. 
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When DAD 330 receives an instruction to display a piece able to dynamically analyze the "best next" button to go to, 
of EMM memory, it consults the MOM data structures to from the current button. This is especially important when 
determine the location of the memory, and the desired considering the fact that the user may modify the look for the 
location of its display. Because DAD 330 is purely BUT menus, on the fly. While the program allows for a fixed 
oriented (e.g., high speed rectangles), the drawing is both 5 transition from button to button, a dynamic method of 
efficient and space saving. Because all display devices have determining the most logical "next choice" foi the user is 
inherent differences, it is DAD*s responsibility to call the implemented as follows. THE engine 340 handles this, with 
appropiate routines at the proper time, for each display help from MOM 300, by keeping track of two cursor 
fl^vice. locations: the current absolute cursor location, and the 

When a new device is added to the system, only a small lO cunent field cursor location. The field cursor marks the 
handful of routines need to be supported for the device. Of current (or most recent) choice executed by the user (e.g., 
the total roster of routines, only routines which the display the last button picked). When a cursor movement command 
device is capable if need be supported. Routines that are not is detected by THE 340, it uses MOM engine 300 to retrieve 
possible for that display device need not be stubbed, and the menu descriptions, and analyzes these descriptions in an 
DAD 330 is simply informed of their absence and handles 15 attempt to determine the best location to naove to, and then 
it gracefully. The routines required embody a small set of performs the move to that location. This form of dynamic 
primitives, which include area blitzing, line blitzing, high sensing affords the most flexibility to the entire application, 
speed rectangle drawing, high speed or outlined rectangle General Handlers for System Independence (GENERAL) 
drawing and cursor support Other routines, not necessarily 350 

supported by all devices include, acquisition, high-speed 20 GENERAL engine 350 handles all routines that arc fairly 
unpacked blitting and automatic packing and unpacking standard, but that for one reason or another are desirable to 
routines, have under platform indqiendent control of the application. 

Because the amount of code for support of a driver is This includes items such as non-EMM engine memory 
relatively small, the drivers are con:^iled directly into the acquisition and freezing, as well as file I/O. GENERAL 
software. This capability allows DAD 330 to actually main- 25 engine 350 is present to give a simple and global mechanism 
tain and provide output to multiple devices simultaneously, for error control and handling. This provides the software 
making it attractive for multiple displays as well as switch- package with a uniform mechanism for errOT recovery, 
ing between hardware on the fly, whidi otherwise may be handled in a less than elegant way 

In order to make DAD 330 handle the platform indepen- by the operating system of run-time program, 
dent drivers, the software calls a standardized set of routines, 30 Signal Processing Engine (IGGY) 360 
which arc table dispatched at runtime. Because of the table IGGY engine 360 is responsible for all digital signal 
dispatch method, no comparisons are required to locate a processing of acquired wave-foma (Le., image) dau. IGGY 
routine within a driver. Instead, all execution is handled by 360 is commanded primarily by MOM 300 and the fiinc- 
hashed indexing Aat provides near-instantaneous access to tionality used to perform some form of wave-form process- 
all drivers an ail functions. Note that because DAD 330 is 35 ing (usually image processing). Instead of being an inte- 
responsible for both input and output display devices, it grated engine (as are MOM 300, RA320, DAD 330, and the 
handles any video acquisition required by video data soft- other engines), IGGY360 is a set of separate routines, which 
ware within the user' s procedure prograra This requires that may bt used and combined to perform high level processing 
DAD 330 coordinate with SIS 380 and MOM 300 at the of acquired data. 

proper time, for control and decoding of incoming video. 40 Due to the nature of the application, in tiiat the end user 
Input Device Engine (THE) 340 does not possess the education necessary to perform high- 

THE engine 340 is designed to be a device-independent end signal processing, ICjGY 360 is designed to have heu- 
input event handler. THE 340 is invoked by both MOM 300 ristic processes that allow it to analyze wave-fom data and 
and the user's functioiiality, to retrieve spatial and textual determine the best enhancements necessary. IGGY 360 can 
information from the user. Because the software is input 45 use MOM 300 to request minimal information from the user, 
device independent, THE 340 is responsible for the conibi- but in general does not do so except in the simplest of cases, 
nation of multiple pointing devices 58, and can handle more IGGY 360 embodies at least the following basic signal 
than one input device simultaneously, as well as being able processing techniques: 
to switch input device context on the fly, 1, Histogram Analysis 

THE engine 340 is basically divided into two sections: 50 2. Histogram Stretching 
textual and spatial. The textual handler involves reading 3 Histogram EquaUzation 
textual data from keyboard devices and transmitting them, Analvsis 
when requested, to MOM 300 and functional software. ^'^ 7 „ ^ f 
Textual handling includes not only input and maintenance of 5. Noise ReducUon 
text buffers, but also flushing of all textual queues. 55 6. Area of Interest Location 

Spatial handling is performed by determining the location 7. Unwanted Area Removal— i.e., rather than compress 
and types of input devices. Because input devices fall into the data, only meaningful data (such as the picture of 

two categories, absolute and relative, THE engine 340 the surgical site) is retained, while other data (such as 

automatically converts all co<^dinate, regardless of their borders that contain no real picture informiation) are 

type, into absolute coordinates. These coordinates are auto- 60 discarded, 
matically dipped, by THE engine 340, against the current Sharpening 
screen resolution as defined by DAD engine 330. 9 Blurring 

One aspect of THE engine 340 is its coupHng with MOM Feature Extraction 

300 for mouseless handling of menu structures. Because " „ uiohhchtina 

MOM 300 aUows menus and buttons to exist anywhere on 65 ^carare mgmignimg 

the screen, even layered, the software that allows the user to 12. Blending 
move from *1ield" to "field" (including buttons) must be 13. Artifact Removal 
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14. Non-liacar balancing Refening also to FIG. 19, when the user invokes program 

15. Colorization. 14 at login time, the user is greeted by an opening screen and 
Though IGGY 360 provides these services in an on-call requested to enter a user name and password. Once entered, 

manner, the analysis services utilize heuristic techniques to these elements are cross compared, via SIS 3 SO, against the 

detennine whether the other servioes are required to provide 5 valid users in the database. When a valid user is identified, 

the best ou^ut data possible. his user profile is loaded, and MOM 300 is informed of any 

Output Device Engine (POP) 370 changes to the environment 

POP engine 370 is to the ou^ut devices as DAD 330 is if ^as an associated procedure form with his or 

to the input devices. For each ou^ut device supported, POP Yiet login, the user is asked whether this is a procedure. If the 

370 provides interface mechanism for th^d^^^^ ^^^^ ^ affirmative, procedure flow control 260 is fol- 

a^so provides bmtedii^t from some of these devices, as ^^^^ otherwise, information flow control 250 is f oUowed. 

the device requires. POP 370 is responsible for maintaining Th.„.l,„ 171™,, ^««*„t '•-ca 

output (and sometimes input) to thVfollowing devices: ^^5"^^ , . 

, , , J i5 1 J I.-. V Once procedure flow control 260 IS accessed, the proce- 

t n ^ccoraers ^ ^ plays it to the user. This is some fonn, entered for or by the 

3. Data Recorders user, when the user is given original access to the software. 

4. Tape Drives j^ic form that is assigned to the user may be modified at any 

5. Disk Drives (hard and floppy) time via the information processor. 

6. FLASH cards (PCMCIA) Once the user fills out die form, the procedure flow 

7. Networks 20 control 260 is launched. This causes DAD 330 to invoke 

8. Modems acquisition and control is switched to the user. THE engine 
It can be seen from die above list that POP 370 is 340 monitors a special remote control, used during proce- 

rtsponsible for some diverse device handling. POP 370 dures (or, optionally, the keyboard), and causes various 

places the devices into "like*" categories, and provides a video functions to be executed. These include, but arc not 

uniform mechanism for calling 'like'' devices. For example, 25 limited to: 

all drives, flash cards and networks arc driven in a like 1, Zoom IN and OUT of the live video image 

manner, while modems have their o.j^ mecha^ as do 2. Freezing the Uve video signal, and then discarding it 

pnnters and film recorders. Smce MOM 300 directs the ^ • i. j , j *u • 

^(put activities. POP 370 is ncva confused as to how the 3. Freezi^the hve video signal, and then saving it 

service is intended (e.g., each service is spedalized, and thus 30 sequentially via PUf jvu 

POP knows the appropriate need and duty). ^- Reviewing all currently acquired video signals 

While most of handling provided by POP 370 requires 5. Executing a TOUR (directed c^ture sequence) 

simple interfacing to outside drivers, or the operating 6. Adjustment of the input video parameters 

system, the printer and film recorder interfaces of POP 370 Captured images are sequentially numbered and named, 

are highly particular to the device being addressed. 3S and are stored on the current mass storage device via POP 

Furthermore, because of all the possible formats for output 370. At the end of the procedure, SIS 380 is automatically 

devices, POP 370 interfaces with MOM 300 to provide the called to store all, or only approved images into the database, 

user with a visual mechanism for the specification of output associated with the current fonn. 

format For example, under printer interface, POP 370 can During the REVIEW process (122, FEG. 6), the user may 

direct MOM 300 to display a grid of images, and have the 40 review all captured images, and optionally delete unwanted 

user . select the appropriate image and its location. images or invoke special image processing on selected 

Furthermore, POP 370 interfaces with RA 320 to allow the images. Images selected for special image processing are 

user to annotate the images prior to output. POP 370 handles processed automatically and overwritten with the new 

interpolation of incorrectly sized data, and can utilize results. When images are overwritten with the new results, 

in-racmory fonts as well as printer provided fonts, depend- 4S an undo file is created, which allows the user to restore the 

ing on die hardware capabilities. original image file. 

Database Engine (SIS) 380 Upon exiting procedure 260, the user is given the ability 

SIS engine 380 is the database handler. It is responsible to download the acquired data to a removable media device 

for handling all database activities that include user (e.g., FLASH memory, etc.), and then is logged off the 

management, forms management, data management and 50 program 

signal processing management Information Processing Control 250 

User management is invoked by MOM 300 to confirm a Information handler 250 is invoked upon successful login, 

user and to set a users profile for the display of MOM menu s. whcD the user dthc^ docs not have an associated (ffoccdurc. 

Forms management handles the design, storage and retrieval or does not wish to access procedure. Once in the informa- 

of forms. Data management handles the storage and query of 55 tion processing system, the user is first greeted with a list of 

data from within forms. Signal processing management is a available forms (via SIS 380). The usn- may select an 

subset of data management and is responsible for the existing form, or select a "new form" option to create a new 

handling of signal data within a form. fonn. If the user opts to create a new fonn. the various form 

Overview of Flow Control elements that are available arc shown to the user, plus the 

As discussed above, program 14 is designed to be a 60 ability to create new form elements. A form element is any 

specific medical application, even though the engines pro- item when exists on any other form, 

vide a much greater, and universal capability. When the user picks an existing form, the form is loaded 

The overriding flow control is a section of the software with blank data. The user may fill out one or more of the 

that dictates to MOM 300, POP 370. SIS 380. and IGGY360 fields in the fc»m, and citiicr enter the form into the database 

a particular method to follow for user flow control. It is this 65 (e.g.. enter the instance of the new data) or else request a 

overriding flow control that provides much of the automatic query. When a qacry is requested, SIS 380 will attempt to 

handling that is required by the naive user. fuliill the missing elements of the form, based on the fields 
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that have been filled in. Upon retrieving the list of available 
responses, the user is shown tiie list and allowed to pick an 
item from the list. Once picked. SIS 380 will load the fonn 
with the appropriate data, and will display it to the user. 
Images, or other signal processing data within the form, 
when selected, will invoke a image browsing menu that 
displays all associated images with the form. Rirther selec- 
tion of an image will cause IGGY engine 360 to be accessed 
for fiill signal processing by the user. 

Information processing procedure 250 also allows the 
user to format and output query responses to the printer, in 
a variety of looks, aU handled by POP 370. 

Finally, for system administrative users, information pro- 
cessing procedure 250 allows new users to be added to the 
system, and existing users to be removed from the system. 
Database Configuration 

Referring to FIG. 21, preference database 20 and all 
image databases are organized in a manner that maximizes 
the flexibility of adding, modifying, or deleting information 
from the databases at will, without rebuilding or restricting 
availability of data. The structure of the databases is rela- 
tively simple, which keeps the data model independent of 
features of host data storage. In addition, database subsets 
(datasets) can be derived, extracted and manipulated 
remotely and then merged back. 

Each data clement is stored independently in database 
storage. That is. there is no record relative pointer structure 
in the datasct. All "pointers" are data relative. One advan- 
tage of this arrangement is that if a database file is damaged, 
pointer chain corruption (which could damage other stored 
data) is avoided. 

A dataset may be transformed into plain text to allow for 
portability. 

Two methods are used to store data: 

System oriented data such as user accounts and associated 
settings, attached device configurations (printers and video), 
and other such information is stored in standard **flat" files. 

Patient data records are stored in a relational database 
designed specifically for the present system. The database is 
designed to meet the following criteria: 

1) host machine independent; 

2) unlimited data types; 

3) ability to add/delete data types during run time; 

4) variable-length, co-existing records; 

5) speed of access regardless of the number of records; 

6) ability to extract a sub-database, modify that subset, 
and micrge the data back into the host database while 
retaining complete relational integrity; 

7) no data space is used for fields that contain no data; 

8) all data (including numeric data) is stored as character 
strings allowing quick, consistent query operations. 

*Nvhat/Its" Design Elements: 
The database consist of just two record types: 

1) the 'Nvhat** records that hold the definitions describing 
the Idnds of data in the datasct; 

2) the *lts" records that contain the acmal data along with 
information on the type of daU and the owner of that 
data (another "Its" record). 

Implementation: 

**what/Its** is implemented as a "layer*' on top of an 
existing database. This allows for maximum flexibility and 
portability to future computing platfornis. In one 
implementation, the commercial database product "Raima" 
is the host database. 

Here is the "what/Its" schema as implemented in 
^Tlaima": 
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«*«*•*••*•«•*«**•«««««*««>* 

database Whatflts { 

data file ''dmuiun.dat" contains wbat,its; 

key 61c *'dimsumkcy'* contains what__id,whaL_owncr, 

its_Jd, it4_o wncr,its_wbat4ts_tag; 
key 61e "dimEumJdt^ cootains its.dau; 
record what { /•field types*/ 

unique key loog what_id; /♦field key*/ 

key bi^g what_owii«r, /•key of its owner*/ 
char what_Jiainc[3 2] ; /♦field name*/ 
long whaLJo^; 
int what_^uiiqi»; 



/♦it's appearance*/ 
/•it's a unique field*/ 
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} 

record its { 

unique key long its_id; 
key bag iti_owner. 



/•data item*/ 
/♦its key id*/ 
♦key id of its owner*/ 



key long its_what /♦ what kind of data is it?*/ 
key char it&_tag; ♦query flag*/ 

key char it4_data(64]; /♦the data itself^/ 



Record the Field Descriptions: 
*'what" reccrds 

These hold the field *'what" deflnitions describing the 
kinds of data in the dataset 
*'what_Jd" field 

is a (long) generated when a new '"what" is instanced. It 
wiU always be a number that is unique to the dataset 
*'what_Jd" is used as the internal key within a dataset used 
to specify what an 'it" record*s "data" field record contains 
(i.c "Last Name " "Patient Address"). 
'Vhat_Xook" field 

contains a (long) whose value is used internally by 
dimsum to define how the data field is interpreted (i.e., 
string4nteger,datejmagefile) 
**what_J^ame" field 

contains the ascii text name of a field type. It is used when 
choosing a field when defining a form. This field is filled 
when a new field type is created at form definition. 
'Its" record 

These records contain the actual dataset along with infor- 
mation on the 'Vhat" of the data audit's owner (as a pointer 
to another **it" record). 
"its_key" field 

contains a (long) that is unique among "its_Jccy" fields 
within the dataset It is tiic true and ultimate identifier of the 
data item in a dataset. 

References to "its_^y" occur in two other fields in an 
"it" record. 

1. If the "its_owner" field is non-0, it will be a "its_Jccy" 
to the data*s owner "it" record. 

2. If "its_type" is 0 (specifying a reference record), the 
"its__data" field will contain a 'its-key" in asdi format. 
This allows one-to-many relations within the dataset 

"its-owner" field 

contains a (long) "its_Jccy" that points to the data's owner 
"it" record, 
"its-what" field 

contains a (long) "what_Jd" that specifies the nature of 
60 what is contained in the "its_data" field. 
"its_data" field 

is an ascii string that holds the data, "fts-data" might 
optionally contain an asdi representation of a *lts-key" 
defining a reference "it" record. 

In the specific implementation described above* data is 
limited to a maximum of 32 bytes for one specific data 
element. Any length may be used. 
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Images and free-form text files arc used in operation of the 
inveatioii. Instead of storing the image and text data in the 
database, only the file name containing the data is stored in 
'Vhat/Ds" 

ANSI "C Code Example: 5 

This is an exan^le routine showing how the '^what/Its'* 
database is iised. The example given here shows how a 
record might be added to the database. 
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-continued 



} 



RoutiiK: SlS^^ter 

Descriptioa: Add a What/lb record to tbe database 
Syntax: F=SIS__catct(striii£s,wbat5^auQt); 
char "sttiugs = Ust of fields w/qaeries 
long what5[ ] a list of what struts are 
int count = # of sniivg^whats pairs 
ixA r = 2 if DO key existed to aUov enter of a new record 
r = 1 if lecozd already existed and no cuter took place 
r = 0 if recoid enter was successful 



10 } 
} 



4_rtcpead((char*)&i£$^_J0B) ;/»get existing*/ 

fDr(k=0,k<count; k++) { /♦set the owner list*/ 
d_koyfiijd(WHAI_JD,&whats[fc],SIS_JDB*/*get each 

what*/ 

<L_reciead((char*)&what^_J}B /♦if this 

owner*/ 

if(whats[j)=rwhat.what_owiicr) 
owoerlist[k}=dt8.its^id; 
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ixA SIS_jen(er(cb3r *strings[ l,k)Qg wbat5[ ],fflt count) 
{ 

int Dokcy,iJJc,dcepcst; 
loogk^what; 
char key[6S]; 
Dokcy=l; 

fOT<i=0-3«i»unt;t++) { 

if(SIS_what_iflL_imique(whflts[i])) { 
sticFy(key,stxings[i]); 
layv/haX = whats{i); 
Dokey^O; 
break; 
} 

} 30 
if(Dokey)tietum(2); /*no key . . . caa'l create! •/ 
if(d_Jwyfials(rrS_J)XrAJcey,SISLJ)B) = SIS_JJOTFOUND) { 

£oto notdoey; 
}elsc{ 

Dok^ = 1; 

while(l) { 

d_jwTead((char»)&its,SIS_JDB); /'get existing •/ 
if^t3.it3__what = keywhat) { 
nokey =0 
break; 

} 

if<d_Jtcymaa(rrS_WH>a;SIS_-DB) = SIS_NOTFOUND) break; 
} 

if<!Dokey) retuin(l); 

} 

notakey: 

for<i=0;i<MAXQp^-^-) { 
t^glist[i] = 0; 
ownerlistfi) = 0; 

} 



TOP_ahow(**Ncw recoid crcated.",OIL_COLOR); 
rctiini(0); 

> 

Record Structure 

Dataset storage is implemented using two record types, 
the 'Nvhat" record 400 and the "it" record 410. *Vhat" 
records 400 describe the data dictionary, while "it" records 
410 contain the actual data. 

Consider the following sample schema definition (in this 
case^ using the RAIMA database): 
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record what ( 

unique key btig what— id; 
key long what_owiier; 
char what^«ne[32 ]; 
long what_Jook; 

} 

record it { 

unique key long its_i£l; 
key long it^owner; 
key long it5_what; 
key char it^data[64]; 

} 



/•field typos*/ 
/♦add key*/ 
/•key of its owner*/ 
/•field name*/ 
/•it's appearance^/ 

/•data item*/ 
/•its key id*/ 
/♦key id of its owoer*/ 
/♦what kind of data b it?*/ 
/•the data itself*/ 
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deepset = 0; 

fbj<F=0;<a>unf4++) { /*ckar buffers*/ 

t»glisl[i] = SIS_what_dcpth(whatsp); /•get it's 

deptii*/ 

if(deepest<tagUst[i] doepc5t=tagtist[i]; 50 
/•reniember deepest*/ 

} 

far<i=0:k>=deepcst;t++) { /*thiu the depths*/ 

fOT^pOy^^unt-jH) { /'do all that are 

this deep*/ 

i£[taglistlj]=i) { /'this is one at our level*/ j ^ 

ifi[ISIS_what_ia_umqiic(whats[i])H /•HERE*/ 
it5jts_id =0; /*its_kl*/ 
it&it5_owoea- ~ owDerlist[j]; /*its owner*/ 

its.it»-wl»t = wbatslil; /•it:L_what»/ 
5ticpy(tl5.it<L-daTa .irtring501)y*ita_data/* 
SISLjCTcateLjt$( ); /•store to db*/ 

}el»{ 

iftd_icyfini(irs_JDAlA,5tringsIjJ^_J)B =SlSLJiCnFOUND) { 
its.it5_id =0; /•il5_id*/ 
itsit^owDcr = ownerlistO]; /♦its_ovner*/ 
itait5_wbat = whatsOI; /•its_what*/ 
sticpy(its.its_dala,stringsljl); /•its_data*/ 
SI£|_creat^it6( ); /*store to db*/ 6^ 

}clse{ 
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Each 'Vhat" record 400 and "it" record 410 has a ran- 
domly generated numeric key that has no special signifi- 
cance other than that it is unique to that specific record in the 
database. 

The data dictionary (i.e., the mechanism that describes the 
types of data contained in the dataset) is held In '*whar 
records 400. For example, a single address book data set 
could require just two * Vhat" records 400 — one f^r a name 
and one for an address. The actual address book data is held 
in **ir records 410. One *'it** record 410 is used fca: each 
name, and one "it^ record 410 stores each address. 

If the dataset needs to be expanded to include^ for 
exaiiq)le, the telephone number of each address entry, a new 
"what^ record 4iM> is created for the telephone nuniber. 
Then, any new entries in the database can also include an 
associated telephone number — but note that the existing data 
is not impacted at all. Consider the following example layout 
of such an address book database: 

The **what" recwds 400 are: 

A. 1024:0:name:l 1024 is the random key; 0 is owner 
(none); "name" is the text name of the field; 1 denotes 
alpha string. 

B. 3046:lQ24:address:l 3046 is the key; 1024 denotes 
that address is associated with "name"; "address" is the 
text name of the field; 1 again denotes an alpha string. 

C. 4096:1024^elq3honc numbcr:2 4096 is die key; 1024 
indicates that telephone is also associated with "name"; 
^telephone number" is the text name of the field; 2 
denotes this is telephone number field. 

The address book entry **it" records 410 arc: 
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1234:0:1024:JohD Doc 

5748:1234:3046:911 Main St. 

8674:1234:4096:(312) 555-1212 

Id the first it record 410, 1234 is the random key of the 
record. 0 shows this record has no associated "owner " 1024 
indicates that this is a "name'* record, and "John Doe" is the 
name data. 

In the second it record 410, 5748 is the random key, 
associates this record with the first (John Doe) it record 410, 
3046 indicates that this is an address record, and **911 Main 
St." is the address data. 

Finally, in the tiiird it record 410, 8674 is the random key, 
1234 associates this record with **John Doe," 4096 denotes 
that this is a telephone number, and *X312) 555-1212" is the 
phone number data. 

The what/it structure allows several query methods. To 
find every data item associated with "John Doe" we search 
the "it" records data fields for "John Doe," read it's key field 
(1234), and then find all of the records that have 1234 in the 
owner field. 

To obtain a list of phone numbers, we simply scan the "it" 
records locking for "4096" in the **its_what" field. If we 
need the name associated with these phone nmubers, we 
look at the "its_owner" field (in John Doe*s case it would 
be 1234) then locate records with 1234 in the "its^id" 
field-and-1024 in the "its_what" field. 

The what/it structure allows several types of data rela- 
tionships to exist including onc-to-onc, onc-to-many, many- 
to-one and many-to-many associations. All of these can be 
created dynamically singly by defining the appropriate 
"what" records. 

Also note that no storage is taken for non-existent data. In 
the address book cxan^le, if there is no telephone number 
for a given entry, then no "it" record is allocated. 

The what/it method also lets us extract a subset of records 
by first obtaining all of the *"what** records and then obtain- 
ing all of the desired "it" records by the "its_what" fields. 
Preference System: 

The preference system integrates system-available 
resources with user-specified preferences, spedalty-specific 
preferences, or procedure- specific preferences. Entry onto 
the data entry fonns, or logging on as a user results in 
configuration of the operating environnient Information 
such as sex, age, left or right side are used by the system so 
that the operator is not presented with options tiiat would be 
excluded given other dioices already known to the system. 
The preference database does more than control only the 
image c£^ture routines. It sends commands to any connected 
devices that cause them to behave in a specified manner. The 
Controls are implemented at 3 levels that are hierarchical so 
that the most specific preference specification will be a 
procedure-related specification, which builds on a specialty- 
specific or user-specific preference, all of which are subject 
to system availability of resources. 

The system determines the total operating range for a user: 
Le., a user can only use what's there. The system ovaiides 
user preferences with a substitution if the user selects an 
unavailable option. The system sets the available range of 
operation for each device. Le., it determines what dioices 
are there for the given device. 

Setup of the system occurs on two levels: 1) setup of the 
system, and 2) setup of a user (typically registered on the 
removable media). 
Setup of the System: 

When a system that has not been previously configured is 
turned on, the system looks for devices that can identify 
themselves to the software program. Some devices (e.g.. 
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video subsystem 34jf) can be identified by the software 
program. Other devices (e.g., an external video camera) may 
not report to the software progranL The interface provides a 
list of possible connected devices (printers, cameras, storage 
5 devices), specification of the identity of the installation (site 
name) and access to set the system date and time clock. 
Options regarding operation of devices can be selected at 
two levels. For devices such as video cameras that have 
generic classes (e.g., RGB input, Y/C input), option selec- 
10 tion is done by selection of the class of devices given in the 
first option. For devices that have specific control options, 
selection of the device brings up a second menu of user 
choices that are preset to the system specified defaults. Once 
selected, these options form the basis for the connected 
15 devices. This may result in changes in the coramands that 
initialize the system. Changes made in the setup will remain 
in force until the commands arc overridden by a user 
preference, ox the setup procedure is repeated. Date and time 
functions are used to set the system dock and will be 
20 maintained by the system dock. Internal accounting is 
performed for both daylight savings time and a perpetual 
calendar. The site information is transferred to individual use 
sessions of this instance of the invention where the indi- 
vidual data form calls for recording of site information. 
23 While this setup information can reside on any storage 
device, in one embodiment, this information is stored on the 
system storage drive, which is normally not accessible to 
user accounts. 
Setup of a User: 
30 Setup of a user presents a form that contains fields for 
entry of user login identity, primary and secondary 
password, user-specific identification, specialty, storage 
device and other preferences. The login identity, primary, 
secondary passwords, specialty and storage device selec- 
35 tions are required entry. The login id and passwords are 
stored to the specified user account In one embodiment, this 
information is stored on the removable media. Hie spcdalty 
selection copies data entry forms and preferences, and an 
empty data structure from the system accessible files to a 
40 user accessible file. Further user preferences or modifica- 
tions arc copied to the uscr*s account. The information 
copied occupies only a small amount of storage space on the 
user's media so that this division of storage is practical. This 
division of storage allows a single user to use many 
45 instances of the machine. Because user-specific information 
is stored on ^e removable media, system maintenance and 
system storage requirements are controlled, and reliability is 
enhanced. 

In the event that operation preferences stored in the user 
50 account result in requests of the system that may not be 
physically possible, the system contains a substitution table 
that allows the system to substitute the most reasonable 
alternative to the user choice given available system 
resources. If no suitable substitute is available, the requested 
55 option is sinq)ly not allowed However, all data is stored in 
such a manner that even if access to a feature is denied, when 
the same user signs on to a system that has the necessary 
resources, the system will utilize them as specified by the 
user preference. 
60 Forms arc the final place where preference is 
inq>lemented, and system and user data integrated. Forms 
serve both as tools to obtain user keyboard input of data, and 
director of operations for the system. Sdection of specialty 
loads the specialty specific forms for procedures commonly 
65 performed by the selected specialty, and containing data 
fields for the information commonly associated with the 
procedure(s) that will be documented using the form. The 
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fonn also contains the specification for the tour scripts that 
will be used during the procedure. The forms also contain 
selections for right left, male and female that may alter the 
tours, or prints of the images* and available selections such 
that only choices related to the correct side and site will be 3 
available until a new form is selected or the side, sex or site 
infonnation is changed (e.g., on a hysterectomy, all pdnts 
will show only female anatomy, and male will not be an 
appropriate choice, whereas on a hernia, male oi female will 
be both necessary and relevant). System and user informa- lo 
tion including date, time, user id, name and site are auto- 
matically entered into the form. Sequence of operations, and 
availability of system options, and how they are presented to 
the user can also be spediied by the forms. The forms also 
contain instructions for how the cursor will traverse the form 15 
(Lc,, where the cursor will move after a given keypress). 

Video preferences, remote control operation specification, 
or ottier parameters may be attached to a given form. As 
sudi, the form is the final entry point prior to starting a 
procedure. Completion of the form is required before pro- 20 
cednres can be initiated. 
Setup of Forms: 

In genaal, setup of a user will result in access to all forms 
normally required by the general user. The system contains 
a capability to modify or design forms. The form builder is 25 
considered an advanced feature, and can only be selected by 
users with ^propriate permissions. The foim builder con- 
sists of a set of tools to specify the appearance of the form, 
and a visual means to specify the actions of die form. The 
user can specify data fields, length, content aiid attribute of 30 
fields (text, a^ha. numeric, validation range), and can 
specify what field the cursor should go to next whctha a 
field is required 01 optional, and relationship, if any, to other 
fields in the form (i.e., right excludes left, male excludes 
female etc.), and the position of the field on the form* The 35 
user can also specify a new tour or series of tours associated 
with the form. This is done by sequential use of the tour 
aeating function. Each sequential image prompt is desig- 
nated. The filing of the tour images is handled by the system 
automatically. 40 

The forms can also call on output layout files, such diat a 
form will specify how images captures while using the form 
will be fonnatted by the sy stenL In general, these formats are 
stared on the system storage devices, and used by the forms. 
The ou^ut layouts are constructed such that the form builder 45 
can substitute information from the form into existing lay- 
outs referenced by a new form. Alternatively, new ou^ut 
layouts can be created and added to system storage to 
support the fonn. If this is done, the layout will be attached 
to the user account, since it will not be available in the 50 
general release. 

The ability to modify forms is dependent on the What/its 
data structure. In a typical relational database structure, 
changes in forms wpuld require an update of the database 
schema. Records created using the old schema may not be 55 
accessible to the new schema and visa versa. With the 
what/its structure, modifications of the form and addidons or 
deletions of fields arc tolerated without compatibility prob- 
lems. Forms themselves are stored as data files that contain 
pointers to the appropriate long array that are used to 60 
construct them. Data is stored in a schema independent layer 
as data itself. The database schema, and key management is 
independent of the data. In one eiubodinient Raima data 
manager is selected for the underlying key management 
scheme, and hard database schema because of its speed in 65 
retrieving data stored with multidirectioDal pointers that the 
what/its structure in^lics. 
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The presence of tours, and associated printing formats 
necessitates a method for managing selection of the carrect 
form when a query is performed For example, a general 
surgeon has forms for L^aroscopic hysterectomy, cholecys- 
tectomy and a generic form. A query for patient name John 
Smith yields a list of matches that have images and data 
related to specific forms. The system stores a hidden mark 
that associates a given patient's data with the correct form 
type. On sdectioo of a given record, the system fetches and 
displays the data to the user on the proper form, even if it is 
different than the form used to initiate the query. 

Other embodiments the invention are within the scope 
of the following daims. For example, while system 10 has 
been described with reference to endoscopy, the system may 
also be used for other medical procedures, surgical or 
otherwise. 
What is claimed is: 

1. A system for acquiring images during a medical pro- 
cedure and using the acquired images, coni^rising 

at least one input device for obtaining said images, 
at least one output device for using the images obtained by 
said input device to enable image data to be commu- 
nicated to a medical practitioner, 
a preference database for pre-storing, for each one of a 
plurality of users of said system, respective preference 
information that indicates one or more processing 
operations that said one of said plurality of users of said 
system prefers to be performed on said images obtained 
by said input device and that indicates a configuration 
of at least one of said input and ou^ut devices that said 
one of said plurality of users of said system prefers, and 
wherein said preference information further indicates, 
for each one of said users, a sequence of images that are 
to be obtained during said medical procedure, and 
wherein said system further comprises means for 
prompting said current user to obtain the images in said 
sequence, and 

a processor for receiving an indication of an identity of 
one of said users, who is currently using said system, 
and, responsive to said indication of said identity of 
said one of said users who is currently using said 
system, for retrieving said preference information for 
said one of said users pre-stored in said preference 
database, for performing processing operations on said 
images obtained by said input device and applying said 
images to said at least one output device based on the 
preference informadon in said preference database that 
corresponds to said current user, and for establishing a 
configuration of at least one of said input and output 
devices based on the preference infonnation in said 
preference database that conesponds to said cunent 
user. 

2. The system of daim 1 wherein said information further 
indicates, for each one of said users, a configuration of said 
at least one input device, and further comprising means for 
establishing said configuration in response to said identity of 
said current user. 

3. The system of claim 2 wherein said configuration 
indudcs a format in which said at least one input device 
produces said image, 

4. The system of daim 1 wherdn said information further 
indicates, for each one of said users, a configuration of said 
at least one output device, and further comprising means for 
establishing said configuration in response to said identity of 
said cunent user. 

5. The system of claim 4 wherein said configuration 
indudcs a format in which said at least one output device 
uses said images. 
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6. The system of claim 1 wherein said preference database 
means to store said information for a plurality of surgical 
procedures, said processor being further responsive to an 
identification of a current one of said surgical procedures 
and performing said processing based on said information 5 
that cQcrespbnds to said current surgical procedure. 

7. The system of claim 1 further comprising a plurality of 
said input devices, said preference database storing said 
information for each one of said ii^)ut devices, said proces- 
sor selectively processing images from said input devices 
based on the identity of said current user and said infoima- 
tion. 

8. The system of claim 1 further comprising a plurality of 
said output devices, said preference database storing said 
information for each one of said output devices,, said pro- 
cessor selectively applying said processed images to said 
ou^ut devices based on the identity of said current user and 
said information. 

9. The system of claim 1 wherein said preference database 
stores said information as a plurality of linked records, a first 
one of said records identifying a type of said information, 20 
and a second one of said records containing data associated 
with said type of information. 

10. The system of claim 1 wherein said at least one output 
device includes means to enable said image data to be 
communicated to said medical pracdtion^ by displaying 23 
said image data. 

11. The system of claim 1 wherein said at least one output 
device includes means to enable said image data to be 
communicated to a medical practitioner by staring said 
image data for later retrieval. 30 

12. A system for acquiring images during a medical 
procedure and using the acquired images, comprising 

at least one input device for obtaining said images, 

at least one output device for using the images obtained by 
said input device to enable image data to be commu- 35 
nicated to a medical practitioner, 

a preference database for pre-storing, for each one of a 
plurality of users of said system, respective preference 
information that indicates a script of designated images 
that said one of said plurality of users of said system 40 
prefers to have captured, and 

a processor for receiving an indication of an identity of 
one of said users, who is currently using said system, 
and, responsive to said indication of said identity of 
said one of said users who is currently using said 
system, for retrieving said preference information for 
said one of said users pre-stored in said preference 
database, and for causing a script of images to be 
captured based on the preference information in said 
preference databaise that corresponds to said current 
user, and said processor including proit^ting means for ^ 
causing said script of images to be captured by pron^t- 
ing a physician to capture said images in accordance 
with said script 

13. A system for acquiring images during a medical 
procedure and using the acquired images, comprising 

at least one input device for obtaining said images, 

at least one output device for using the images obtained by 
said input device to enable image data to be conunu- 
nicated to a medical practitioner, 

a preference database for pre-storing, for each one of a ^ 
plurality of medical procedures, respective preference 
information that indicates a script of designated images 
that a user of said system prefers to have raptured in 
connection with said one of said plurality of medical 
procedures, and 65 

a processor for receiving an indication of an identity of 
one of said plurality of medical procedures, and. 
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responsive to said indication of said one of said plu- 
rality of medical procedures, for retrieving said pref- 
erence information for said one of said medical proce- 
dures pre-stored in said preference database, and for 
causing a script of images to be captured based on the 
preference information in said preference database that 
corresponds to said one of said medical procedures, and 
said processor including prompting means far causing 
said script of Lmages to be captured by prompting a 
physician to capture said images in accordance with 
said script 

14. A system for acquiring images during a medical 
procedure and using the acquired images, comprising 

at least one input device for obtaining said images, 

at least one output device f using the images obtained by 
said input device to enable image data to be coimnu- 
nicated to a medical practitioner, 

a preference database for pre-storing, for each one of a 
plurality of users of said system, respective preference 
information that indicates one or more visual represen- 
tations that said one <rf said plurality of us^s of said 
system prefers to be applied to said at least one output 
device together with said images obtained by said input 
device, said one or more visual representations com- 
prising graphical objects, and 

a processor for receiving an indication of an identity of 
one of said users, who is currently using said system, 
and, responsive to said indication of said identity of 
said one of said users who is currently using said 
system, for retrieving said preference information for 
said one of said users pre-stored in said preference 
database, for combining said one or more visual rep- 
resentations with said images obtained by said input 
device based on the preference infcmnation in said 
preference database that corresponds to said one of said 
users, and for applying said images to said at least one 
output device together with said one or more visual 
r^resentations. 

15. A system in accordance with daim 14 wherein said 
one or mof e visual representations comprise text 

16. A system in accordance with claim 15 wherein said 
processor includes means to anti-alias said text for improved 
appearance. 

17. A system in accordance with claim 14 wherein said 
one or more visual representations conqjrises at least one 
additional image. 

18. A system in accordance with claim 17 wherein said 
processor includes means to obtain said additional image 
from an additional input device. 

19. A system in accordance with daim 17 wherein said 
images applied to said at least one ou^ut device conoprise 
images that said processcx" has caused to be stored in a 
memory upon rece^)t of a save command during said 
medical procedure. 

20. A system in accordance with daim 19 wherein said 
processor includes means to cause said at least one output 
device to display said images as a composite of postage 
staii^ size linages. 

21. A system in accordance witfi daim 19 whttcin said 
processor includes means to cause said at least one output 
device to display said images as a sequence of full size 
images. 

22. A system in accordance widi daim 19 wherein said 
processor includes means to invoke image processing on 
images sdected by said user firom said images implied to 
said at least one output device. 

23. A system in accordance with claim 19, wherein said 
processor includes means to delete an unwanted image 
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selected by said user from said images applied to said at least 
oae output device. 

24. A system in accordance with claim 14 wherein said 
visual rq)TeseDtatLon possesses the attribute of translucency. 

25. A system in accordance with claim 14 wherein said 5 
one or more visual representations con^nisc prompts for 
prompting said one of said users to capture images in a 
desired sequence. 

26. A system in accordance with claim 14 wherein said 
processor includes means to maintain an aspect ratio of said 
one or more visual rq)resentations with respect to said 
images as said images are scaled in size. 

27. A system in accordance with claim 14 wherein said 
processor includes means to maintain relative appearance of 
said one or more visual representations with respect to said 
images as a format of said images is changed 

28. A system in accordance with daim 14 wherein said 
processor is configured to store said one en* more visual 
representations in overlays in order not to affect said images. 

29. A system for acquiring images during a medical 
procedure and using the acquired images, conqirising 20 

at least one input device for obtaining said images^ 

at least one output device for using the images obtained by 
said input device to enable image data to be commu- 
nicated to a medical practitioner, 

a preference database for pre-storing, for each one of a 2s 
plurality of medical procedures, re^ective preference 
mformation that indicates one or more visual represen- 
tations that are preferred to be applied to said at least 
one output device togetho* with said images obtained 
by said ii^ut device in connection with said one of said 
plurality of medical procedures, said one or more visual 
rqiresentations comprising graphical objects, and 

a processor for receiving an indication of an identity of 
one of saidmcdical procedures, whidi is currently to be 
performed and, responsive to said indication of said 
identity of said one of said medical procedures that is 35 
currently to be performed, for retrieving said prefer- 
ence information for said one of said medical proce- 
dures that is fffc-stored in said preference database, fcr 
combining said one or more visual r^resentations with 
said images obtained by said input device based on the 
preference information in said preference database that ^ 
corresponds to said medical procedure, and for apply- 
ing said images to said at least one output device 
together with said one or more visual representations. 

30. A system in accordance with claim 29 wherein said 
one or more visual representations comprise text 43 

31. A system in accordance with claim 30 wherein said 
processor includes means to anti-alias said text for improved 
qxpearance. 

32. A system in accordance with claim 29 wherein said 
one or more visual representations comprises at least one 50 
additional image. 

33. A system in accordance with claim 32 wherein said 
processor includes means to obtain said additional image 
from an additional input device. 

34. A system in accordance with claim 32 wherein said 55 
images applied to said at least one ou^ut device comprise 
images that said processor has caused to be stored in a 
memory upon receipt of a save conmiand during said 
medical procedure. 

36. A system in accordance with claim 34 wherein said 60 
processor includes means to cause said at least one output 
device to display said images as a composite of postage 
stamp size images. 

36. A system in accordance with claim 34 wherein said 
processor includes means to cause said at least one output 65 
device to display said images as a sequence of full size 
images. 
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37. A system in accordance with claim 34, wherein said 
processor includes means to invoke image processing on 
images selected by a user from said images appHcd to said 
at least one output device. 

38. A system in accordance with claim 34, wherein said 
processor includes means to delete an unwanted image 
selected by said a from said images applied to said at least 
one output device. 

39. A system in accordance with daim 29 wherein said 
visual representation possesses the attribute of translucency. 

40. A system in accordance with claim 29 wherein said 
one or more visual representations comprise prompts for 
pron^ting said one of said users to capture images in a 
desired sequence. 

41. A system in accordance with claim 29 wherein said 
processor includes means to maintain an aspect ratio of said 
one or more visual representations with respect to said 
images as said images are scaled in size. 

42. A system in accordance with claim 29 wherein said 
processor includes means to maintain relative appearance of 
said one or more visual representations with respect to said 
images as a format of said unages is changed. 

43. A system in accordance with daim 29 wherein said 
processor includes means to store said one or more visual 
representations in overlays in order not to affect said images. 

44. A system for acquiring images during a medical 
procedure and using the acquired images, con^sing 

at least one input device fcv obtaining said images, 

at least one output device for using the images obtained by 
said input device to enable image data to be commu- 
nicated to a medical practitioner, 

a still frame buffer configured to temporarily store an 
image obtained by said at least one input device. 

a memory configured to store images obtained by said at 
least one input device on a long-term basis, and 

a processor configured to recdve a freeze command and 
in response thereto to freeze in said still frame buffer an 
image obtained by said at least one input device, and to 
receive a save command and in response thereto to save 
in said memory a frozen image. 

45. A system in accordance with daim 44 wherein said 
processor is further configured to recdve a live command 
and in response thereto to cause said at least one output 
device to display live images obtained by said at least one 
input device, said processor being configured to toggle 
between frozen and live images until a desired image is 
frozen and then saved in said mem<»7. 

46. A system for acquiring images during a medical 
procedure and using the acquired images, con^rising 

at least one input device fca* obtaining said images, 

at least one output device for using the images obtained by 
said input device to enable image data to be commu- 
nicated to a medical practitioner, 

a preference database for pre-storing, for each one of a 
plurality of users of said system, respective preference 
information that indicates one or more visual represen- 
tations that said one of said plurality of users of said 
system prefers to be applied to said at least one output 
device together with said images obtained by said input 
device, said one or more visual representations pos- 
sessing the attributes of translucency. and 

a processor for receiving an indication of an identity of 
one of said users, who is currently using said system, 
and, responsive to said indication of said identity of 
said one of said users who is currently using said 
system, for retrieving said preference information for 
said one of said users pre- stored in said preference 
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database* for combining said one or more visual rep- 
resentations with said images obtained by said input 
device based on the preference information in said 
preference database that corresponds to said one of said 
users, and for applying said images to said at least one ^ 
ou^ut device together with said one or more visual 
representations. 

47. A system for acquiring images during a medical 
procedure and using the acquired images, con^rising 

at least one input device for obtaining said images, 

at least one output device for using the images obtained by 
said input device to enable image data to be commu- 
nicated to a medical practitioner^ 

a preference database for prc-storing, for eadi one of a 
plurality of users of said system, respective preference 
information that indicates one or more visual represen- 15 
tations that said one of said plurality of users of said 
system prefers to be applied to said at least one output 
device together with said images obtained by said input 
device, said one or more visual representations com- 
prising prompts for pron^ting said one of said users to ^ 
capture images in a desired sequence^ and 

a processor for receiving an indication of an identity of 
one of said users, who is cuirently using said system, 
and, responsive to said indication of said identity of 
said one of said users who . is currently using said 
system, for retrieving said preference information for ^ 
said one of said users pre-stored in said preference 
database, for combining said one or more visual rep- 
resentations with said images obtained by said input 
device based on the preference information in said 
preference database that corresponds to said one of said 30 
users, and for applying said images to said at least one 
output device together with said one or more visual 
representations. 

48. A system for acquiring images during a medical 
procedure and using the acquired images, comprising 35 

at least one input device for obtaining said images, 
at least one ou^ut device for using the images obtained by 
said input device to enable image data to be commu- 
nicated to a medical practitioner, 
a preference database for prc-storing, for each one of a 40 
plurality of users of said system, respective preference 
informatioQ that indicates one or more visual represen- 
tations that said one of said plurality of users of said 
system prefers to be applied to said at least one ou^ut 
device together with said images obtained by said input 
device, 

a processor for receiving an indication of an identity of 
one of said users, who is currently using said system, 
and, responsive to said indication of said identity of 
said one of said users who is currently using said ^ 
system, for retrieving said preference information for 
said one of said users pre-stored in said preference 
database, for combining said one or more visual rep- 
resentations with said images obtained by said input 
device based on the preference information in said 
preference database that corresponds to said one of said 
users, and for applying said images to said at least one 
output device together with said one or more visual 
representations, and 

said processor including means for maintaining an aspect 
ratio of said one or more visual representations with 60 
respect to said images as said images are scaled in size. 

49. A system for acquiring images during a medical 
procedure and using the acquired images, comprising 

at least one input device for obtaining said images, 
at least one output device for using the images obtained by 65 
said input device to enable image data to be commu- 
nicated to a medical practitioner. 
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a preference database for pre-storing, for each one of a 
plurality of users of said system, respective preference 
infonnation that indicates one or more visual represen- 
tations that said one of said plurality of users of said 
system prefers to be applied to said at least one output 
device together with said images obtained by said input 
device, 

a processor for receiving an indication of an identity of 
one of said users, who is currently using said system, 
and, responsive to said indication of said identity of 
said one of said users who is currently using said 
system, for retrieving said preference information for 
said one of said users pre-stored in said preference 
database, for combining said one or more visual rep- 
resentations with said images obtained by said input 
device based on the preference information in said 
preference database that corresponds to said one of said 
users, and for f^lying said images to said at least one 
ou^ut device together with said one or more visual 
representations, and 

said processor including means for maintaining relative 
appearance of said one or more visual representations 
with respect to said images as a format of said images 
is dianged. 

50. A system for acquiring images during a medical 
procedure and using the acquired images, comprising 

at least one input device for obtaining said images, 
at least one output device for using the images obtained by 
said input device to enable image data to be commu- 
nicated to a medical practitioner, 
a preference database for pre-stoiing, for each one of a 
plurality of users of said system, respective preference 
information that indicates one or more visual represen- 
tations that said one of said plurality of users of said 
system prefers to be applied to said at least one output 
device together with said images obtained by said input 
device, 

a processor for receiving an indication of an identity of 
one of said users, who is currently using said system, 
and, responsive to said indication of said identity of 
said one of said users who is currently using said 
system, for retrieving said preference information for 
said one of said users pre-stored in said preference 
database, for combining said one or more visual rep- 
resentations widi said images obtained by said input 
device based on the preference information in said 
preference database that corresponds to said one of said 
users, and for applying said images to said at least one 
output device together with said one or more visual 
representations, and 

said processor including means for storing said one or 
more visual representations in overlays in order not to 
affect said images. 

51. A system for acquiring images during a medical 
procedure and using the acquired images, comprising 

at least one input device for obtaining said images, 

at least one output device for using the images obtained by 
said input device to enable image data to be commu- 
nicated to a medical practitioner, 

a preference database for pre-storing, for each one of a 
plurality of medical procedures, respective preference 
information that indicates one or more visual represen- 
tations that are preferred to be applied to said at least 
one ou^ut device together with said images obtained 
by said input device in connection with said one of said 
plurality of medical procedures, said one or more visual 
representations possessing the attributes of 
translucency, and 

a processor for receiving an indication of an identity of 
one of said medical procedures, which is currently to be 
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performed and, responsive to said indication of said 
identity of said one of said medical procedures that is 
currently to be perfctrmed, for retrieving said prefer- 
ence infomiation for said one of said medical proce- 
dures that is prc-stored in said preference database, for 5 
combining said one or more visual representations with 
said images obtained by said input device based on the 
preference information in said preference database that 
coiresponds to said medical procedure, and for apply- 
ing said linages to said at least one output device 
together with said one or more visual representations. 

52. A system for acquiring images during a medical 
procedure and using the acquired images, con^)rising 

at least one input device for obtaining said images, 
at least one output device for using the images obtained by j 3 
said input device to enable image data to be commu- 
nicated to a medical practitLoner, 
a preference database for pre-storing, for eadi one of a 
plurality of medical procedures, respective preference 
information that indicates one or more visual represen- 20 
tations that arc prcfared to be applied to said at least 
one ou^ut device together with said images obtained 
by said input device in connection with said one of said 
plurality of medical procedures, said one or more visual 
representations comprising prompts for prompting said 
one of said users to capture images in a desired 
sequence, and 

a processor for receiving- an indication of an identity of 
one of said medical procedures, which is currently to be 
performed and, responsive to said indication of said 
identity of said one of said medical procedures that is 
currently to be perfc^med, for retrieving said prefer- 
ence information fen: said one of said medical proce- 
dures that is pre-storedin said preference database, for 
combining said one or more visual representations with 
said images obtained by said input device based on the 35 
preference information in said preference database that 
c<»responds to said medical procedure, and for apply- 
ing said images to said at least one output device 
together with said one or more visual representations. 

53. A system for acquiring images during a medical 40 
procedure and using the acquired images, comprising 

at least one input device for obtaining said iniages, 
at least one output device for using the images obtained by 
said input device to enable image data to be commu- 
nicated to a medical practitioner, 4S 
a preference database for pre-storing, for cadi one of a 
plurality of medical procedures, respective preference 
information that indicates one or more visual represen- 
tations that are preferred to be applied to said at least 
one output device together with said images obtained 50 
by said iiiput device in connection with said one of said 
plurality of medical procedures, and 

a processor for receiving an indication of an identity of 
one of said medical procedures, which is currently to be 
performed and, responsive to said indication of said 55 
identity of said one of said medical procedures that is 
currently to be performed, for retrieving said prefer- 
ence information for said one of said medical proce- 
dures that is pre-stored in said preference database, for 
combining said one or more visual representations with ^ 
said images obtained by said input device based on the 
preference information in said preference database that 
corresponds to said medical procedure, and for apply- 
ing said images to said at least one output device 
together with said one or more visual representations, 
and 



said processor including means for maintaining an aspect 
ratio of said one or more visual representations with 
respect to said images as said images are scaled in size. 

54. A system for acquiring images during a medical 
procedure and using the acquired images, comprising 

at least one input device for obtaining said images, 

at least one output device for using the images obtained by 
said input device to enable image data to be commu- 
nicated to a medical practitioner, 

a preference database for pre-storing, for each one of a 
plurality of medical procedures, respective preference 
information that indicates one or more visual represen- 
tations that arc preferred to be applied to said at least 
one output device together with said images obtained 
by said input device in connection with said one of said 
plurality of medical procedures, and 

a processor for receiving an indication of an identity of 
one of said medical procedures, whidi is currently to be 
peiformed and, responsive to said indication of said 
identity of said one of said medical procedures that is 
currently to be performed, for retrieving said prefer- 
ence information for said one of said medical proce- 
dures that is pre-storcd in said preference database, for 
combining said one or more visual representations with 
said images obtained by said input device based on the 
preference information in said preference database that 
coiresponds to said medical procedure, and for apply- 
ing said images to said at least one output device 
together with said one or more visual representations, 
and 

said processor including means for maintaining relative 
appearance of said one or more visual representations 
with respect to said images as a format of said images 
is changed. 

55. A system for acquiring images during a medical 
procedure and using the acquired images, comprising 

at least one input device for obtaining said images, 

at least one output device for using the images obtained by 
said input device to enable image data to be commu- 
nicated to a medical practitioner, 

a preference database for pre-storing, for each one of a 
plurality of medical procedures, respective preference 
information that indicates one or more visual represen- 
tations that are preferred to be applied to said at least 
one ou^ut device together with said images obtained 
by said input device in connection with said one of said 
plurality of medical procedures, and 

a processor for receiving an indication of an identity of 
one of said medical procedures, which is currently to be 
performed and, responsive to said indication of said 
identity of said one of said medical procedures that is 
currently to be performed, for retrieving said prefer- 
ence infonnation for said one of said medical proce- 
dures that is prc-stored In said preference database, fox 
conabining said one or more visual representations with 
said images obtained by said input device based on the 
preference information in said preference database that 
corresponds to said medical procedure, and for apply- 
ing said images to said at least one output device 
together with said one or more visual representations, 
and 

said processor including means for storing said one or 
more visual representations in overlays in order not to 
affect said images. 
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